Re: [openstack-dev] [TripleO] Plan management refactoring for Life cycle

[mathieu bultel]

Hi,

On 09/11/2018 12:08 PM, Bogdan Dobrelya wrote:
> On 9/11/18 4:43 AM, James Slagle wrote:
>> On Mon, Sep 10, 2018 at 10:12 AM Jiri Tomasek 
>> wrote:
>>>
>>> Hi Mathieu,
>>>
>>> Thanks for bringing up the topic. There are several efforts
>>> currently in progress which should lead to solving the problems
>>> you're describing. We are working on introducing CLI commands which
>>> would perform the deployment configuration operations on deployment
>>> plan in Swift. This is a main step to finally reach CLI and GUI
>>> compatibility/interoperability. CLI will perform actions to
>>> configure deployment (roles, networks, environments selection,
>>> parameters setting etc.) by calling Mistral workflows which store
>>> the information in deployment plan in Swift. The result is that all
>>> the information which define the deployment are stored in central
>>> place - deployment plan in Swift and the deploy command is turned
>>> into simple 'openstack overcloud  deploy'. Deployment plan
>>> then has plan-environment.yaml which has the list of environments
>>> used and customized parameter values, roles-data.yaml which carry
>>> roles definition and network-data.yaml which carry networks
>>> definition. The information stored in these files (and deployment
>>> plan in general) can then be treated as source of information about
>>> deployment. The deployment can then be easily exported and reliably
>>> replicated.
>>>
>>> Here is the document which we put together to identify missing
>>> pieces between GUI,CLI and Mistral TripleO API. We'll use this to
>>> discuss the topic at PTG this week and define work needed to be done
>>> to achieve the complete interoperability. [1]
>>>
>>> Also there is a pending patch from Steven Hardy which aims to remove
>>> CLI specific environments merging which should fix the problem with
>>> tracking of the environments used with CLI deployment. [2]
>>>
Thank you Jirka to point me to this work.
I will be happy to help in those efforts, at least for the lice cycle
part (Update/Upgrade/Scale) of this big feature. I can't attend to the
PTG this week unfortunately, but if you can point me out the etherpad
with the resume of the session it would be very nice.

I think the review from Steven aim to solve more or less the same issue
than my current review. I will go through it in details, and AFAIS the
last changes are old.

>>> [1]
>>> https://gist.github.com/jtomasek/8c2ae6118be0823784cdafebd9c0edac
>>> (Apologies for inconvenient format, I'll try to update this to
>>> better/editable format. Original doc:
>>> https://docs.google.com/spreadsheets/d/1ERfx2rnPq6VjkJ62JlA_E6jFuHt9vVl3j95dg6-mZBM/edit?usp=sharing)
>>> [2] https://review.openstack.org/#/c/448209/
>>
>>
>> Related to this work, I'd like to see us store the plan in git instead
>> of swift. I think this would reduce some of the complexity around plan
>> management, and move us closer to a simpler undercloud architecture.
>> It would be nice to see each change to the plan represented as new git
>> commit, so we can even see the changes to the plan as roles, networks,
>> services, etc, are selected.
>>
>> I also think git would provide a familiar experience for both
>> developers and operators who are already accustomed to devops type
>> workflows. I think we could make these changes without it impact the
>> API too much or, hopefully, at all.
>
> +42!
> See also the related RFE (drafted only) [0]
>
> [0] https://bugs.launchpad.net/tripleo/+bug/1782139

Thanks James,
Same here +1 (or 42 :))
>
>>
>
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] Plan management refactoring for Life cycle

[mathieu bultel]

Hi folks,

Last week I wrote a BluePrint and a spec [1] to propose to change the
way we used and managed the Plan in TripleO for the Deployment and the
Life cycle (update/upgrade and scale).

While I was working on trying to simplified the implementation of the
Update and Upgrade for a end user usage, I found very hard to follow all
the calls that the TripleO Client was doing to the HeatClient and
SwiftClient.

I traced the calls and found that we can safely and easily decrease the
number of calls and simplified the way that we are computing & rendering
the TripleO Heat Templates files.

I did a PoC to see what would be the problematic behind that and what we
could do without breaking the "standard" usage and all the particular
things that the current code handle (specific deployments and
configurations & so on).

By this refactoring I'm seeing another gain for the life cycle part of
TripleO, where we used to try to make thing simpler & safer but we
constantly failed due to this complexity and all the "special cases"
that we faced during the testing.

The result is that, when a user need to perform an update/upgrade of his
deployment, he really have to be careful, to pay a lot of attention of
all the options, -e environments files that he previously used  with the
risk to make a simple mistake, and totally mess up the deployment.

So my goals with this PoC and this BP is to try to addressed those
points by:

simplify  and reduce the number of calls between the clients,

have a simple way for creating and updating the Plan, even by
amending the plan with only a particular files / config or Playbooks,

store all the in formations provided by the user by uploading all
the files outsides of the plan,

keep the track of the environment files passed to the CLI,

trace the life cycle story of the deployment.

So feel free to comment, add your concerns or feedback around this.

Cheer,

Mathieu

[1]

https://blueprints.launchpad.net/tripleo/+spec/tripleo-plan-management

https://review.openstack.org/599396 

[2]

 https://review.openstack.org/583145


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] quickstart for humans

[mathieu bultel]

Hi

On 08/30/2018 04:28 PM, Honza Pokorny wrote:
> Hello!
>
> Over the last few months, it seems that tripleo-quickstart has evolved
> into a CI tool.  It's primarily used by computers, and not humans.
> tripleo-quickstart is a helpful set of ansible playbooks, and a
> collection of feature sets.  However, it's become less useful for
> setting up development environments by humans.  For example, devmode.sh
> was recently deprecated without a user-friendly replacement. Moreover,
> during some informal irc conversations in #oooq, some developers even
> mentioned the plan to merge tripleo-quickstart and tripleo-ci.
>
> I think it would be beneficial to create a set of defaults for
> tripleo-quickstart that can be used to spin up new environments; a set
> of defaults for humans.  This can either be a well-maintained script in
> tripleo-quickstart itself, or a brand new project, e.g.
> tripleo-quickstart-humans.  The number of settings, knobs, and flags
> should be kept to a minimum.
>
> This would accomplish two goals:
>
> 1.  It would bring uniformity to the team.  Each environment is
> installed the same way.  When something goes wrong, we can
> eliminate differences in setup when debugging.  This should save a
> lot of time.
>
> 2.  Quicker and more reliable environment setup.  If the set of defaults
> is used by many people, it should container fewer bugs because more
> people using something should translate into more bug reports, and
> more bug fixes.
>
> These thoughts are coming from the context of tripleo-ui development.  I
> need an environment in order to develop, but I don't necessarily always
> care about how it's installed.  I want something that works for most
> scenarios.
>
> What do you think?  Does this make sense?  Does something like this
> already exist?
I'm agree with the fact that quickstart has turned into a CI tool, more
than a human tool.
I still use quickstart to deploy and work on TripleO but I feel a bit
lost when I have to grep into the config/ dirctory to see which
featureset match to my needs and, if not, try to tweak the config and
pray that the tweaked options will work as expected.

I also discovered the ci reproducer script recently. The script probably
need to be a bit more robust, but it's a real gain when you have to
reproduce CI environments.

I think a first less effort for now would be to bring a set of
Quickstart commands and "human readable config files" to improve the
situation.

> Thanks for listening!
>
> Honza
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Upgrade community weekly meeting

[mathieu bultel]

Hi All,

After few weeks and meetings on Friday, we decide to change a little bit
the pattern of this meeting.
We decided to keep the schedule up and open for anything/anyone, but
each times, we will check before the meeting if the agenda [1] is empty.
If so, then we won't make it, or we would use this schedule for another
topic.
This has been decided like that for few reasons. The main reason is that
we repeat and discuss the same status/issues that we previously discuss
on our normal scrum and most of the attendees is folks from inside the
DFG. So its a kind of a repetition for us and there is no big value in
this except when people external of the DFG bring things, like Weshay
used to or rdo-cloud folks.

Thank you all.
Mathieu

[1]

https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting



On 11/09/2017 01:23 PM, mathieu bultel wrote:
> Hi TripleO,
>
> As discuss on the last TripleO meeting, the Upgrade squad will stand a
> weekly meeting each Friday to discuss and share about the Upgrade
> related topics.
>
> Every body who are interested is welcome to join us and fell free to
> raise any questions/issues/concerns.
>
> Tomorrow will be our first session and it will be more CI oriented.
>
> We will use this etherpad to track the discussions and the status:
>
> https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting
>
> For the next meetings, I will setup agenda for each of those with the
> following topics: Bugs, CI and Features (Specs), but everybody can add
> his own items in the agenda.
>
> Mathieu
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Upgrade community weekly meeting

[mathieu bultel]

Hi All,

After few weeks and meetings on Friday, we decide to change a little bit
the pattern of this meeting.
We decided to keep the schedule up and open for anything/anyone, but
each times, we will check before the meeting if the agenda [1] is empty.
If so, then we won't make it, or we would use this schedule for another
topic.
This has been decided like that for few reasons. The main reason is that
we repeat and discuss the same status/issues that we previously discuss
on our normal scrum and most of the attendees is folks from inside the
DFG. So its a kind of a repetition for us and there is no big value in
this except when people external of the DFG bring things, like Weshay
used to or rdo-cloud folks.

Thank you all.
Mathieu

[1]

https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting



On 11/09/2017 01:23 PM, mathieu bultel wrote:
> Hi TripleO,
>
> As discuss on the last TripleO meeting, the Upgrade squad will stand a
> weekly meeting each Friday to discuss and share about the Upgrade
> related topics.
>
> Every body who are interested is welcome to join us and fell free to
> raise any questions/issues/concerns.
>
> Tomorrow will be our first session and it will be more CI oriented.
>
> We will use this etherpad to track the discussions and the status:
>
> https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting
>
> For the next meetings, I will setup agenda for each of those with the
> following topics: Bugs, CI and Features (Specs), but everybody can add
> his own items in the agenda.
>
> Mathieu
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Proposing Wesley Hayutin core on TripleO CI

[mathieu bultel]

On 12/06/2017 04:45 PM, Emilien Macchi wrote:
> Team,
>
> Wes has been consistently and heavily involved in TripleO CI work.
> He has a very well understanding on how tripleo-quickstart and
> tripleo-quickstart-extras work, his number and quality of reviews are
> excellent so far. His experience with testing TripleO is more than
> valuable.
> Also, he's always here to help on TripleO CI issues or just
> improvements (he's the guy filling bugs on a Saturday evening).
> I think he would be a good addition to the TripleO CI core team
> (tripleo-ci, t-q and t-q-e repos for now).
> Anyway, thanks a lot Wes for your hard work on CI, I think it's time
> to move on and get you +2 ;-)
>
> As usual, it's open for voting, feel free to bring any feedback.
> Thanks everyone,

+1 of course


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Proposing Ronelle Landy for Tripleo-Quickstart/Extras/CI core

[mathieu bultel]

+1 indeed :)

On 11/29/2017 08:34 PM, John Trowbridge wrote:
> I would like to propose Ronelle be given +2 for the above repos. She
> has been a solid contributor to tripleo-quickstart and extras almost
> since the beginning. She has solid review numbers, but more
> importantly has always done quality reviews. She also has been working
> in the very intense rover role on the CI squad in the past CI sprint,
> and has done very well in that role.
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo]

[mathieu bultel]

On 11/07/2017 11:47 PM, Emilien Macchi wrote:
> On Wed, Nov 8, 2017 at 9:29 AM, Wesley Hayutin  wrote:
>> Greetings,
>>
>> I'd like to propose we remove the upgrade jobs that are consistently failing
>> from the upstream infrastructure and instead focus our efforts in RDO
>> Software Factory.
>>
>> The jobs listed in https://review.openstack.org/#/c/518405/ are consistently
>> failing after being reviewed by myself and Mathieu.  I am leaving
>> legacy-tripleo-ci-centos-7-multinode-upgrades in place as it's passing with
>> an overall rate of 87%.
>>
>> It doesn't make any sense to continue to tax upstream resources on failing
>> jobs, lets get the jobs running correctly and consistently in rdo software
>> factory before moving these back to our mainline CI.
>>
>> Please let me know what you think of the proposal.
> +1 to remove them if we have the scenario upgrades (with parity from
> what we had upstream) in RDO CI.
> We'll need to make the job experimental in RDO CI, so we can only run
> them at demand until they actually work. Can we do that as well?
>
> Thanks,

+1,

I missed this thread due to no subject ;)

Thank you Wes for having moved those jobs in SF and for raising this thread.

Mathieu


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] Upgrade community weekly meeting

[mathieu bultel]

Hi TripleO,

As discuss on the last TripleO meeting, the Upgrade squad will stand a
weekly meeting each Friday to discuss and share about the Upgrade
related topics.

Every body who are interested is welcome to join us and fell free to
raise any questions/issues/concerns.

Tomorrow will be our first session and it will be more CI oriented.

We will use this etherpad to track the discussions and the status:

https://etherpad.openstack.org/p/tripleo-upgrade-squad-meeting

For the next meetings, I will setup agenda for each of those with the
following topics: Bugs, CI and Features (Specs), but everybody can add
his own items in the agenda.

Mathieu


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] plans on testing minor updates?

[mathieu bultel]

Hi,


On 09/28/2017 05:05 AM, Emilien Macchi wrote:
> I was reviewing https://review.openstack.org/#/c/487496/ and
> https://review.openstack.org/#/c/487488/ when I realized that we still
> didn't have any test coverage for minor updates.
> We never had this coverage AFICT but this is not a reason to not push
> forward it.
Thank you for the review and the -2! :)
So I'm agree with you, we need CI coverage for that part, and I was
wondering how I can put quickly a test in CI for the minor update.
But before that, just few things to take in account regarding those reviews:

1/ Those patches are needed for Pike and we are pretty (pretty, pretty)
late.
The reviews was implemented 2 or 3 weeks ago, but we have made a lot of
tests with both, dev and QE environments (QE more complex and realistic
than dev env or even CI env, Ceph nodes, multi computes and controllers)
to be sure to have something clearly working with less bugs as possible.
I think it is..

2/ All those patches are touching code which is not (and never) tested
by CI at all... which is bad, but Rome was not built in one day, right ?
;)  No job for config download, no job for minor update, no job for ...
Yes I can iterate, there is a lot of features in TripleO without CI
coverage.

3/ Config download code has no CI tests at all except unit tests of
course and the minor update "core" feature has been already implemented
and merged. Those reviews are "only" CLI implementations.

4/ I tried to push unit tests on all parts of the reviews, I think it's
an acceptable tests status for now, to get this landed. Unit tests can
be, in some cases, more relevant than big CI (integration) tests.

5/ Why, instead of blocking the reviews, not make a follow up review
with the CI coverage ? I know it would be better to make it now, or even
early, but I think it can be sane to just create a blocker LP and
implement the workflow for Master.

6/ In the mean time, I think we need to work on a workflow for the
future features to implement regarding CI.
Can someone from the CI squad can help to implement new features ? Or
new features only belong to the DFG which creates it ?  (If so, I would
say for Upgrades, hey guys we don't care about upgrading your stuffs, do
it yourself and fix your bugs ;))

So I understand the concerns but my worries here is that this feature is
needed for Pike and implementing a new job now, will take very long time
and add more delay for the workflow to have it in P.
If the target was for Queens, I would say "yes, lets push a great CI
coverage for this feature".

Can we make a consensus ?

> During Ocata and Pike, we saw that having upgrade jobs were extremely
> useful to actually test the workflow that our users are supposed to do
> in production, I see zero reason to not doing the same for minor
> updates.
> I don't want to be the bad guy here but i've -2 the 2 patches until we
> find some consensus here (sorry matbu, it's not against you or your
> code in specific, but more generally speaking about re: implementing
> features without CI coverage).
>
> I'm really willing to help and start to work on tripleo-quickstart
> roles this week, if someone agrees to pair with me - so we could make
> progress and have that coverage. Even if the new job would fail,
> that's OK we know the process might work (or not, TBH, I haven't tried
> it, probably shardy and some other folks know more about it). Once we
> have the workflow in place, then iterate into matbu's patches and make
> it work in CI so we can ship it and be proud to have the feature
> tested.
> That's IMHO how we should write our software.
>
> If there is any feedback on this, please let us know here, otherwise
> I'll keep my -2 until we've got this coverage in place. Also please
> someone (maybe matbu?) raise your hand if you want to pair up and do
> this quickly.
>
> Thanks,



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] weekly meetings on #tripleo

[mathieu bultel]

On 07/18/2017 11:22 PM, Emilien Macchi wrote:
> On Mon, Jul 17, 2017 at 12:10 PM, Emilien Macchi  wrote:
>> Since we have mixed feelings but generally agree that we should give
>> it a try, let's give it a try and see how it goes, at least one time,
>> tomorrow.
> So we tried and did the meeting on #tripleo.
> I noticed more people participated and were present to the meeting and
> I didn't notice any interruption.
>
> Please bring any feedback (positive / negative) so we decide if
> whether or not we continue that way.
+1 , in particular for those, like me, who are bad with calendar / time
schedule things :)
>> On Mon, Jul 10, 2017 at 10:01 AM, Michele Baldessari  
>> wrote:
>>> On Mon, Jul 10, 2017 at 11:36:03AM -0230, Brent Eagles wrote:
 +1 for giving it a try.
>>> Agreed.
>>>
 On Wed, Jul 5, 2017 at 2:26 PM, Emilien Macchi  wrote:

> After reading http://lists.openstack.org/pipermail/openstack-dev/2017-
> June/118899.html
> - we might want to collect TripleO's community feedback on doing
> weekly meetings on #tripleo instead of #openstack-meeting-alt.
>
> I see some direct benefits:
> - if you come up late in meetings, you could easily read backlog in
> #tripleo
> - newcomers not aware about the meeting channel wouldn't have to search
> for it
> - meeting would maybe get more activity and we would expose the
> information more broadly
>
> Any feedback on this proposal is welcome before we make any change (or
> not).
>
> Thanks,
> --
> Emilien Macchi
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> --
>>> Michele Baldessari
>>> C2A5 9DA3 9961 4FFB E01B  D0BC DDD4 DCCB 7515 5C6D
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> --
>> Emilien Macchi
>
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] Validations before upgrades and updates

[mathieu bultel]

On 05/08/2017 01:45 PM, Marios Andreou wrote:
> Hi folks, after some discussion locally with colleagues about
> improving the upgrades experience, one of the items that came up was
> pre-upgrade and update validations. I took an AI to look at the
> current status of tripleo-validations [0] and posted a simple WIP [1]
> intended to be run before an undercloud update/upgrade and which just
> checks service status. It was pointed out by shardy that for such
> checks it is better to instead continue to use the per-service
>  manifests where possible like [2] for example where we check status
> before N..O major upgrade. There may still be some undercloud specific
> validations that we can land into the tripleo-validations repo
> (thinking about things like the neutron networks/ports, validating the
> current nova nodes state etc?).
Yes, I think a bunch of validation: db states, services states, network
connectivity (external, internal)
>
> So do folks have any thoughts about this subject - for example the
> kinds of things we should be checking - Steve said he had some reviews
> in progress for collecting the overcloud ansible puppet/docker config
> into an ansible playbook that the operator can invoke for upgrade of
> the 'manual' nodes (for example compute in the N..O workflow) - the
> point being that we can add more per-service ansible validation tasks
> into the service manifests for execution when the play is run by the
> operator - but I'll let Steve point at and talk about those. 
>
I have a WIP review about that [1], but i need to revisit it a bit, to
add a part into the mistral workflow (Im also writing a POC to create a
mistral workbook for major upgrade and validate minor/major upgrade
before starting [2], I have a third one in progress, not pushed yet, to
implement the major upgrade option in the cli):

[1] https://review.openstack.org/444224
[2] https://review.openstack.org/#/c/462961


> cheers, marios
>
> [0] https://github.com/openstack/tripleo-validations 
> [1] https://review.openstack.org/#/c/462918/
> [2]  
> https://github.com/openstack/tripleo-heat-templates/blob/stable/ocata/puppet/services/neutron-api.yaml#L197
>  
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [tripleo] [tripleo-quickstart] pending reviews for composable upgrade for Ocata

[mathieu bultel]

Hi,

I'm sending this email to the list to request reviews about the
composable upgrade work I have been done in Tripleo quickstart. It's
pending for a while (dec 4 for one of those 2 reviews), and I have
addressed all the comments on time, rebase & so one [1].
Those reviews is required, and very important for 3 reasons:
1/ It addressed the following BP: [2]
2/ It would give a tool for the other Squad and DFGs to start to play
with composable upgrade in order to support their own components.
3/ It will be a first shot for the Tripleo-CI / Tripleo-Quickstart
transition for supporting the tripleo-ci upgrade jobs that we have
implemented few weeks ago now.

I updated the documentation (README) regarding the upgrade workflow, the
commit message explain the deployment workflow, I know it's not easy to
review this stuff, and probably tripleo-quickstart cores don't give
importance around this subject. I think I can't do much more now for
making the review more easy for the Cores.

It was one of my concerns about adding all the very specific extras
roles (upgrade / baremetal / scale) in one common repo, loosing flexibly
and reaction, but it's more than that...

I'm planning to write a "How To" for helping to other DFGs/Squads to
work on upgrade, but since this work is still under review, I'm stuck.

Thanks.

[1]
tripleo-quickstart repo:
https://review.openstack.org/#/c/410831/
tripleo-quickstart-extras repo:
https://review.openstack.org/#/c/416480/

[2]

https://blueprints.launchpad.net/tripleo/+spec/tripleo-composable-upgrade-job



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Upstream backwards compatibility job for Newton oooq

[mathieu bultel]

On 01/17/2017 05:19 PM, Emilien Macchi wrote:
> On Tue, Jan 17, 2017 at 10:57 AM, mathieu bultel <mbul...@redhat.com> wrote:
>> On 01/17/2017 04:42 PM, Emilien Macchi wrote:
>>> On Tue, Jan 17, 2017 at 9:34 AM, mathieu bultel <mbul...@redhat.com> wrote:
>>>> Hi Adriano
>>>>
>>>> On 01/17/2017 03:05 PM, Adriano Petrich wrote:
>>>>
>>>> So I want to make a backwards compatibility job upstream so from last scrum
>>>> I got the feeling that we should not be adding more stuff to the
>>>> experimental jobs due to lack of resources (and large queues)
>>>>
>>>> What kind of "test" do you want to add ?
>>>> I ask because since few days we have upstream an upgrade job that does:
>>>> master UC -> deploying a Newton OC with Newton OC + tht stable/newton ->
>>>> then upgrade the OC to master with tht master branch.
>>>> It's sounds like a "small backward compatibility" validation, but I'm not
>>>> sure if it's cover what you need.
>>> While I understand what is the idea, I don't see the use case.
>>> In which case you want to deploy a old version of overcloud by using a
>>> recent undercloud?
>>> Why don't use deploy a stable undercloud to deploy a stable overcloud?
>> From my side, the use case is the major OC upgrade. We don't want to
>> test the major upgrade of the undercloud (since a job already exist),
>> only overcloud, that's why we start by a "master" undercloud, and that
>> save us from unwanted/unrelated issues due to the UC upgrade and reduce
>> the duration of the job.
> ok so your use-case is CI focused. Good to know.
> Another question for you then, have we count the time needed to
> upgrade an undercloud? I'm doing it quite oftent and it doesn't take
> more than 15 min for me. Thoughts?
Yes it's approximately this duration. Actually it depend on the hardware
and the network connectivity, but saying 15/20 minutes avg sounds
reasonable.
>
>>>> Is that so? I was thinking about using nonha-multinode-oooq that seems to 
>>>> be
>>>> working.
>>>>
>>>> Is that allright to add this new job or should I wait until we get more
>>>> resource and do ci.centos for now, or any idea on where to do this is also
>>>> welcome.
>>>>
>>>>
>>>> Cheers,
>>>>Adriano
>>>>
>>>>
>>>> __
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>> __
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Upstream backwards compatibility job for Newton oooq

[mathieu bultel]

On 01/17/2017 04:42 PM, Emilien Macchi wrote:
> On Tue, Jan 17, 2017 at 9:34 AM, mathieu bultel <mbul...@redhat.com> wrote:
>> Hi Adriano
>>
>> On 01/17/2017 03:05 PM, Adriano Petrich wrote:
>>
>> So I want to make a backwards compatibility job upstream so from last scrum
>> I got the feeling that we should not be adding more stuff to the
>> experimental jobs due to lack of resources (and large queues)
>>
>> What kind of "test" do you want to add ?
>> I ask because since few days we have upstream an upgrade job that does:
>> master UC -> deploying a Newton OC with Newton OC + tht stable/newton ->
>> then upgrade the OC to master with tht master branch.
>> It's sounds like a "small backward compatibility" validation, but I'm not
>> sure if it's cover what you need.
> While I understand what is the idea, I don't see the use case.
> In which case you want to deploy a old version of overcloud by using a
> recent undercloud?
> Why don't use deploy a stable undercloud to deploy a stable overcloud?
>From my side, the use case is the major OC upgrade. We don't want to
test the major upgrade of the undercloud (since a job already exist),
only overcloud, that's why we start by a "master" undercloud, and that
save us from unwanted/unrelated issues due to the UC upgrade and reduce
the duration of the job.

>
>> Is that so? I was thinking about using nonha-multinode-oooq that seems to be
>> working.
>>
>> Is that allright to add this new job or should I wait until we get more
>> resource and do ci.centos for now, or any idea on where to do this is also
>> welcome.
>>
>>
>> Cheers,
>>Adriano
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Upstream backwards compatibility job for Newton oooq

[mathieu bultel]

Hi Adriano

On 01/17/2017 03:05 PM, Adriano Petrich wrote:
> So I want to make a backwards compatibility job upstream so from last
> scrum I got the feeling that we should not be adding more stuff to the
> experimental jobs due to lack of resources (and large queues) 
>
What kind of "test" do you want to add ?
I ask because since few days we have upstream an upgrade job that does:
master UC -> deploying a Newton OC with Newton OC + tht stable/newton ->
then upgrade the OC to master with tht master branch.
It's sounds like a "small backward compatibility" validation, but I'm
not sure if it's cover what you need.
> Is that so? I was thinking about using nonha-multinode-oooq that seems
> to be working.
>
> Is that allright to add this new job or should I wait until we get
> more resource and do ci.centos for now, or any idea on where to do
> this is also welcome.
>
>
> Cheers,
>Adriano
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][Upgrade Squad syncup]

[mathieu bultel]

On 12/14/2016 10:17 AM, Julie Pichon wrote:
> On 14 December 2016 at 08:56, Dougal Matthews  wrote:
>> On 13 December 2016 at 16:12, Emilien Macchi  wrote:
>>> On Tue, Dec 13, 2016 at 10:58 AM, Marios Andreou 
>>> wrote:
 Hi OOOs o/ as discussed on this week's upstream weekly irc meeting [1]
 lets schedule a syncup on the Ocata composable upgrades work and how we
 will build on Steve Hardy's base ansible driven implementation.

 IMO a call would be the best format (I propose to schedule a bluejeans
 call, though I am open to any other suggestions). For now I've setup a
 doodle poll at http://doodle.com/poll/rp8ck6tstaagftqr - please vote if
 you'd like to participate. Assuming we go with bluejeans (unless I hear
 strong pushback/other suggestions) then I'll try and make a recording
 available to address the concerns expressed at

 http://lists.openstack.org/pipermail/openstack-dev/2016-December/108780.html
 (thanks shardy for pointing that out).

 Moving forward we should probably switch to an irc based syncup,
>>> As long as:
>>> - we have zero pushbash from our community members
>>> - anyone can join the call
>>> - our design and discussions stay open (mailing-list, tripleo-specs, IRC)
>>> - a nice summary is sent to openstack-dev
>> or maybe even better, a recording is posted online?
> Personally, I find summaries easier to parse compared to finding an
> hour to listen to people talking back and forth about something,
> though it doesn't need to be an either/or proposition :) One of the
> issues brought up in the other thread about video calls is that they
> can be difficult to follow for folks who don't have English as a first
> language. Having a few written notes about what came up during the
Hehe, thanks Julie, I have excatly the same feeling, +1 with you :)

> meeting and some of the conclusions would help with this I think.
>
> Looking forward to seeing how the upgrade work comes along!
>
> Julie
>
>>> - we keep tripleo meeting on IRC every week
>>>
>>> I don't see any blocker to do video-conference between folks that work
>>> together on a same topic.
>>>
 I'll close the poll at 10UTC tomorrow morning (sorry for short notice :(
 ) and schedule the meeting accordingly with a followup mail here,


 thanks, marios


 [1]

 http://eavesdrop.openstack.org/meetings/tripleo/2016/tripleo.2016-12-13-14.00.log.html
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] proposing Michele Baldessari part of core team

[mathieu bultel]

On 11/04/2016 06:40 PM, Emilien Macchi wrote:
> MIchele Baldessari (bandini on IRC) has consistently demonstrated high
> levels of contributions in TripleO projects, specifically in High
> Availability area where's he's for us a guru (I still don't understand
> how pacemaker works, but hopefully he does).
>
> He has done incredible work on composable services and also on
> improving our HA configuration by following reference architectures.
> Always here during meetings, and on #tripleo to give support to our
> team, he's a great team player and we are lucky to have him onboard.
> I believe he would be a great core reviewer on HA-related work and we
> expect his review stats to continue improving as his scope broadens
> over time.
>
> As usual, feedback is welcome and please vote for this proposal!
>
> Thanks,
+1  of course ! :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] what's up in upgrades?

[mathieu bultel]

Hi all,

Since we decided to switch on OVB provisioner for the CI, the upgrade
jobs is now working really better and consistently.
Unfortunately, I can answer now if we will reach the timeout with those
jobs because the current state of the upgrade (mitaka to newton) is
currently broken due to a number of issues which we are working on.

I'm making some cleaning in the (huge) review
(https://review.openstack.org/#/c/323750) atm, but if all of you are ok,
it would be great to merge this work, even if the upgrade (full
overcloud) is not complete successfully.
It would allow us, at least, to trigger the experimental pipeline and
see what's going on with the upgrade.

( and example of the state of the full upgrade job:
http://logs.openstack.org/50/323750/54/experimental-tripleo/gate-tripleo-ci-centos-7-ovb-nonha-upgrades-nv/45916e6/console.html)

On 09/09/2016 09:29 AM, mathieu bultel wrote:
> On 09/09/2016 12:42 AM, Emilien Macchi wrote:
>> On Thu, Sep 8, 2016 at 4:18 PM, David Moreau Simard <d...@redhat.com> wrote:
>>> How long does the upgrade job take ?
>> undercloud upgrade is a matter of 10 min max.
>> overcloud upgrade will be much more, I don't have metrics right now.
>> matbu maybe?
> It really depend on the infra which run the upgrade. I don't know much
> about the upstream infra but
> on my local box, with a SSD, 8 cores and 32Go of RAM, It could take
> around 1h30 2hours for a full upgrade.
> On centos ci infra and with RDO, some jobs can takes 4hours or so ...
>
> I'm really curious to see how long a full upgrade will take with upstream.
>
> Right now, the full upgrade job didn't go far from the controller
> upgrade (step 2).
> AFAIK, the timeout in upstream is 3 hours minus 10 minutes ...
> I think if we keep a 2 nodes deployment with only pacemaker, it would be
> enough... I will keep you in touch of my progress here..
>
> But, even if the jobs takes 2 or 3 hours to vote, I think it would be a
> real huge gain for the tripleo work.
>>> David Moreau Simard
>>> Senior Software Engineer | Openstack RDO
>>>
>>> dmsimard = [irc, github, twitter]
>>>
>>>
>>> On Thu, Sep 8, 2016 at 2:27 PM, Emilien Macchi <emil...@redhat.com> wrote:
>>>> What's up in upgrades?
>>>>
>>>> 1) Undercloud: Mitaka to Newton
>>>>
>>>> We just approved a patch in openstack-infra/tripleo-ci that test
>>>> undercloud upgrades.
>>>> I don't think we should make it vote as for now it's quite
>>>> experimental. Though I'm wondering if we should move it to the check
>>>> pipeline as non-voting (currently in experimental queue).
>>>>
>>>> This is a first iteration and if you plan to upgrade your undercloud,
>>>> you'll still have to do manual steps that we do in tripleo-ci. They
>>>> are described here:
>>>> https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577
>>>>
>>>> We need to decide where to put these bits: in tripleoclient? in
>>>> instack-undercloud? Let's talk about it.
>>>>
>>>>
>>>> 2) Overcloud: Mitaka to Newton
>>>>
>>>> matbu and myself are working together on a CI job that will test the
>>>> upgrade of an undercloud + overcloud from Mitaka to Newton.
>>>> Work is here: https://review.openstack.org/#/c/364859 and
>>>> https://review.openstack.org/#/c/323750/ (both patches are going to
>>>> merge together so we have one single patch for review).
>>>> The idea is to use multinode job for now as a first iteration, with
>>>> the simplest scenario possible so we can easily iterate later.
>>>>
>>>>
>>>> 3) Overcloud: Newton to Newton
>>>> I'm working on a simple patch that would test updates from Newton to
>>>> Newton: https://review.openstack.org/#/c/351330/ like we had with OVB
>>>> jobs but this time using multinode.
>>>>
>>>>
>>>> Any feedback, help, is highly welcome.
>>>> --
>>>> Emilien Macchi
>>>>
>>>> __
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] what's up in upgrades?

[mathieu bultel]

On 09/09/2016 12:42 AM, Emilien Macchi wrote:
> On Thu, Sep 8, 2016 at 4:18 PM, David Moreau Simard  wrote:
>> How long does the upgrade job take ?
> undercloud upgrade is a matter of 10 min max.
> overcloud upgrade will be much more, I don't have metrics right now.
> matbu maybe?
It really depend on the infra which run the upgrade. I don't know much
about the upstream infra but
on my local box, with a SSD, 8 cores and 32Go of RAM, It could take
around 1h30 2hours for a full upgrade.
On centos ci infra and with RDO, some jobs can takes 4hours or so ...

I'm really curious to see how long a full upgrade will take with upstream.

Right now, the full upgrade job didn't go far from the controller
upgrade (step 2).
AFAIK, the timeout in upstream is 3 hours minus 10 minutes ...
I think if we keep a 2 nodes deployment with only pacemaker, it would be
enough... I will keep you in touch of my progress here..

But, even if the jobs takes 2 or 3 hours to vote, I think it would be a
real huge gain for the tripleo work.
>> David Moreau Simard
>> Senior Software Engineer | Openstack RDO
>>
>> dmsimard = [irc, github, twitter]
>>
>>
>> On Thu, Sep 8, 2016 at 2:27 PM, Emilien Macchi  wrote:
>>> What's up in upgrades?
>>>
>>> 1) Undercloud: Mitaka to Newton
>>>
>>> We just approved a patch in openstack-infra/tripleo-ci that test
>>> undercloud upgrades.
>>> I don't think we should make it vote as for now it's quite
>>> experimental. Though I'm wondering if we should move it to the check
>>> pipeline as non-voting (currently in experimental queue).
>>>
>>> This is a first iteration and if you plan to upgrade your undercloud,
>>> you'll still have to do manual steps that we do in tripleo-ci. They
>>> are described here:
>>> https://github.com/openstack-infra/tripleo-ci/blob/41e8560cf3d313f2be69df64e4c95a3240dfa402/scripts/tripleo.sh#L554-L577
>>>
>>> We need to decide where to put these bits: in tripleoclient? in
>>> instack-undercloud? Let's talk about it.
>>>
>>>
>>> 2) Overcloud: Mitaka to Newton
>>>
>>> matbu and myself are working together on a CI job that will test the
>>> upgrade of an undercloud + overcloud from Mitaka to Newton.
>>> Work is here: https://review.openstack.org/#/c/364859 and
>>> https://review.openstack.org/#/c/323750/ (both patches are going to
>>> merge together so we have one single patch for review).
>>> The idea is to use multinode job for now as a first iteration, with
>>> the simplest scenario possible so we can easily iterate later.
>>>
>>>
>>> 3) Overcloud: Newton to Newton
>>> I'm working on a simple patch that would test updates from Newton to
>>> Newton: https://review.openstack.org/#/c/351330/ like we had with OVB
>>> jobs but this time using multinode.
>>>
>>>
>>> Any feedback, help, is highly welcome.
>>> --
>>> Emilien Macchi
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] State of upgrade CLI commands

[mathieu bultel]

On 08/18/2016 09:29 AM, Marios Andreou wrote:
> On 17/08/16 15:46, Jiří Stránský wrote:
>> On 16.8.2016 21:08, Brad P. Crochet wrote:
>>> Hello TripleO-ians,
>>>
>>> I've started to look again at the introduced, but unused/undocumented
>>> upgrade commands. It seems to me that given the current state of the
>>> upgrade process (at least from Liberty -> Mitaka), these commands make
>>> a lot less sense.
>>>
>>> I see one of two directions to take on this. Of course I would love to
>>> hear other options.
>>>
>>> 1) Revert these commands immediately, and forget they ever existed.
>>> They don't exactly work, and as I said, were never officially
>>> documented, so I don't think a revert is out of the question.
>>>
>>> or
>>>
>>> 2) Do a major overhaul, and rethink the interface entirely. For
>>> instance, the L->M upgrade introduced a couple of new steps (the AODH
>>> migration and the Keystone migration). These would have either had to
>>> have completely new commands added, or have some type of override to
>>> the existing upgrade command to handle them.
>>>
>>> Personally, I would go for step 1. The 'overcloud deploy' command can
>>> accomplish all of the upgrade steps that involve Heat. In order for
>>> the new upgrade commands to work properly, there's a lot that needs to
>>> be refactored out of the deploy command itself so that it can be
>>> shared with deploy and upgrade, like passing of passwords and the
>>> like. I just don't see a need for discrete commands when we have an
>>> existing command that will do it for us. And with the addition of an
>>> answer file, it makes it even easier.
>>>
>>> Thoughts?
>>>
>> +1 for approach no. 1. Currently `overcloud deploy` meets the upgrade
>> needs and it gave us some flexibility to e.g. do migrations like AODH
>> and Keystone WSGI. I don't think we should have a special command for
>> upgrades at this point.
>>
>> The situation may change as we go towards upgrades of composable
>> services, and perhaps wrap upgrades in Mistral if/when applicable, but
>> then the potential upgrade command(s) would probably be different from
>> the current ones anyway, so +1 for removing them.
> +1 from me too, especially because this ^^^ (the workflow we currently
> have and use will change quite drastically I expect)
>
> thanks, sorry I didn't spot this earlier,
> marios

+1 for me too, even if, I think for an end-user experience it's not
ideal and the CLI would be a better way for this point.
>> Jirka
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev