Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Boden Russell]

On 3/14/18 6:59 PM, Tony Breeds wrote:
> On Thu, Mar 15, 2018 at 07:16:11AM +0900, Akihiro Motoki wrote:
>> (1) it makes difficult to run tests in local environment
>> We have only released version of neutron/horizon on PyPI. It means
>> PyPI version (i.e. queens) is installed when we run tox in our local
>> development. Most neutron stadium projects and horizon plugins depends
>> on the latest master. Test run in local environment will be broken. We
>> need to install the latest neutron/horizon manually. This confuses
>> most developers. We need to ensure that tox can run successfully in a
>> same manner in our CI and local environments.
> 
> This is an issue I agree and one we need to think about but it will be
> somewhat mitigated for local development by pbr siblings[1]
> 
> In the short term, developers can do something like:
> 
> for env in pep8,py35,py27 ; do
>tox -e $env --notest
>.tox/$env/bin/pip install -e /path/to/{horizon,neutron}
>tox -e $env
> done
> 
> Which is far from ideal but gives as a little breathing room to decide
> if we need to revert and try again in a while or persist with the plan
> as it stands.
> 
> pbr siblings wont fix all the issues we have and still makes consumption of
> neutron and horizon (and plugins / stadium projects) difficult outside
> of test.

Unless I'm missing something, devstack is also impacted in these
scenarios and doesn't account for installing master branches of select
dependencies. As a result we are seeing failures in our external CI jobs
(that use devstack) due to invalid package versions.

Is the proper way to address this to specify the _REPO and _BRANCH in
our project's devstack lib sciprt(s) as needed so that devstack will
grab master for them?

Thanks

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Doug Hellmann]

Excerpts from Thomas Morin's message of 2018-03-15 10:15:38 +0100:
> Hi Doug,
> 
> Doug Hellmann, 2018-03-14 23:42:
> > We keep doing lots of infra-related work to make it "easy" to do
> >  when it comes to
> > managing dependencies.  There are three ways to address the issue
> > with horizon and neutron, and none of them involve adding features
> > to pbr.
> > 
> > 1. Things that are being used like libraries need to release like
> >libraries. Real releases. With appropriate version numbers. So
> >that other things that depend on them can express valid
> > dependencies.
> > 
> > 2. Extract the relevant code into libraries and release *those*.
> > 
> > 3. Things that are not stable enough to be treated as a library
> >shouldn't be used that way. Move the things that use the
> > application
> >code as library code back into the repo with the thing that they
> >are tied to but that we don't want to (or can't) treat like a
> >library.
> 
> What about the case where there is co-development of features across
> repos ? One specific case I have in mind is the Neutron stadium where

We do that all the time with the Oslo libraries. It's not as easy as
having everything in one repo, but we manage.

> we sometimes have features in neutron repo that are worked on as a pre-
> requisite for things that will be done in a neutron-* or networking-*
> project. Another is a case for instance where we need to add in project
> X a tempest test to validate the resolution of a bug for which the fix
> actually happened in project B (and where B is not a library).

If the tempest test can't live in B because it uses part of X, then I
think X and B are really one thing and you're doing more work than you
need to be doing to keep them in separate libraries.

> My intuition is that it is not illegitimate to expect this kind of
> development workflow to be feasible; but at the same time I read your
> suggestion above as meaning that it belongs to the real of "things we
> shouldn't be doing in the first place".  The only way I can reconcile

You read me correctly.

We install a bunch of components from source for integration tests
in devstack-gate because we want the final releases to work together.
But those things only interact via REST APIs, and don't import each
other.  The cases with neutron and horizon are different. Even the
*unit* tests of the add-ons require code from the "parent" app. That
indicates a level of coupling that is not being properly addressed by
the release model and code management practices for the parent apps.

> the two would be to conclude we should collapse all the module in
> neutron-*/networking-* into neutron, but doing that would have quite a
> lot of side effects (yes, this is an understatement).

That's not the only way to do it. The other way would be to properly
decompose the shared code into a library and then provide *stable
APIs* so code can be consumed by the add-on modules. That will make
evolving things a little more difficult because of the stability
requirement. So it's a trade off. I think the teams involved should
make that trade off (in one direction or another), instead of
building tools to continue to avoid dealing with it.

So let's start by examining the root of the problem: Why are the things
that need to import neutron/horizon not part of the neutron/horizon
repositories in the first place?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Andreas Jaeger]

On 2018-03-15 10:05, Thomas Morin wrote:
> Hi Andreas, all,
> 
> Andreas Jaeger, 2018-03-14 20:46:
>> Note that thanks to the tox-siblings feature, we really continue to
>> install neutron and horizon from git - and not use the versions in the
>> global-requirements constraints file.
> 
> This addresses my main concern, which was that by removing
> tools/tox_install.sh we would end up not pulling master from git.
> 
> The fact that we do keep pulling from git wasn't explicit AFAIK in any
> of the commit messages of the changes I had to look at to understand
> what was being modified.

Sorry for not mentioning that.

> I concur with Akihiro's comment, and would go slightly beyond that:
> ideally the solution chosen would not only technical work, but would
> reduce the ahah-there-is-magic-behind-the-scene effect, which is a pain
> I believe for many: people new to the community face a steeper learning
> curve, people inside the community need to spend time adjust, and infra
> folks end up having to document or explain more. In this precise case,
> the magic behind the scene (ie. the tox-siblings role) may lead to
> confusion for packagers (why our CI tests as valid is not what appears
> in requirements.txt) and perhaps people working in external communities
> (e.g. [1]).

The old way - included some magic as well ;(

I agree with Doug - we need to architect our dependencies better to
avoid these problems and hacks,

Andreas

> Best,
> 
> -Thomas
> 
> [1]
> http://docs.opnfv.org/en/latest/submodules/releng-xci/docs/xci-overview.html#xci-overview


-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Thomas Morin]

Hi Doug,

Doug Hellmann, 2018-03-14 23:42:
> We keep doing lots of infra-related work to make it "easy" to do
>  when it comes to
> managing dependencies.  There are three ways to address the issue
> with horizon and neutron, and none of them involve adding features
> to pbr.
> 
> 1. Things that are being used like libraries need to release like
>libraries. Real releases. With appropriate version numbers. So
>that other things that depend on them can express valid
> dependencies.
> 
> 2. Extract the relevant code into libraries and release *those*.
> 
> 3. Things that are not stable enough to be treated as a library
>shouldn't be used that way. Move the things that use the
> application
>code as library code back into the repo with the thing that they
>are tied to but that we don't want to (or can't) treat like a
>library.

What about the case where there is co-development of features across
repos ? One specific case I have in mind is the Neutron stadium where
we sometimes have features in neutron repo that are worked on as a pre-
requisite for things that will be done in a neutron-* or networking-*
project. Another is a case for instance where we need to add in project
X a tempest test to validate the resolution of a bug for which the fix
actually happened in project B (and where B is not a library).

My intuition is that it is not illegitimate to expect this kind of
development workflow to be feasible; but at the same time I read your
suggestion above as meaning that it belongs to the real of "things we
shouldn't be doing in the first place".  The only way I can reconcile
the two would be to conclude we should collapse all the module in
neutron-*/networking-* into neutron, but doing that would have quite a
lot of side effects (yes, this is an understatement).

-Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Thomas Morin]

Hi Andreas, all,
> Note that thanks to the tox-siblings feature, we really continue to
> install neutron and horizon from git - and not use the versions in
> the global-requirements constraints file.

This addresses my main concern, which was that by removing
tools/tox_install.sh we would end up not pulling master from git.

The fact that we do keep pulling from git wasn't explicit AFAIK in any
of the commit messages of the changes I had to look at to understand
what was being modified.

I concur with Akihiro's comment, and would go slightly beyond that:
ideally the solution chosen would not only technical work, but would
reduce the ahah-there-is-magic-behind-the-scene effect, which is a pain
I believe for many: people new to the community face a steeper learning
curve, people inside the community need to spend time adjust, and infra
folks end up having to document or explain more.  In this precise
case,  the magic behind the scene (ie. the tox-siblings role) may lead
to confusion for packagers (why our CI tests as valid is not what
appears
in requirements.txt) and perhaps people working in external communities
(e.g. [1]).
Best,

-Thomas

[1] http://docs.opnfv.org/en/latest/submodules/releng-xci/docs/xci-over
view.html#xci-overview

Andreas Jaeger, 2018-03-14 20:46:__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Doug Hellmann]

Excerpts from Tony Breeds's message of 2018-03-15 11:59:00 +1100:
> On Thu, Mar 15, 2018 at 07:16:11AM +0900, Akihiro Motoki wrote:
> > The current version of proposed patches which drops tox_install.sh
> > works in our CI. Even if we have neutron>=12.0.0 (queens) or
> > horizon>=13.0.0 (queens), if we have "required-projects" in zuul v3
> > config, tox-sibling role ensures to install the latest master of
> > neutron/horizon. It is okay in our CI.
> > 
> > On the other hand, this change brings a couple of problems. I think it
> > is worth discussed broadly here.
> > 
> > (1) it makes difficult to run tests in local environment
> > We have only released version of neutron/horizon on PyPI. It means
> > PyPI version (i.e. queens) is installed when we run tox in our local
> > development. Most neutron stadium projects and horizon plugins depends
> > on the latest master. Test run in local environment will be broken. We
> > need to install the latest neutron/horizon manually. This confuses
> > most developers. We need to ensure that tox can run successfully in a
> > same manner in our CI and local environments.
> 
> This is an issue I agree and one we need to think about but it will be
> somewhat mitigated for local development by pbr siblings[1]
> 
> In the short term, developers can do something like:
> 
> for env in pep8,py35,py27 ; do
>tox -e $env --notest
>.tox/$env/bin/pip install -e /path/to/{horizon,neutron}
>tox -e $env
> done
> 
> Which is far from ideal but gives as a little breathing room to decide
> if we need to revert and try again in a while or persist with the plan
> as it stands.
> 
> pbr siblings wont fix all the issues we have and still makes consumption of
> neutron and horizon (and plugins / stadium projects) difficult outside
> of test.
>  
> > (2) neutron/horizon version in requirements.txt is confusing
> > In the cycle-with-milestone model, a new version of neutron/horizon
> > will be released only when a release is shipped.
> > The code depends on the latest master, but requirements.txt says it
> > depends on queens or later. It sounds confusing.
> 
> Yes we either need to create a new release-model or switch
> neutron/horizon to the cycle-with-intermediary model and encourage
> appropriate releases.  I'd really like to avoid publishing daily to pypi.

We keep doing lots of infra-related work to make it "easy" to do
things we shouldn't be doing in the first place when it comes to
managing dependencies.  There are three ways to address the issue
with horizon and neutron, and none of them involve adding features
to pbr.

1. Things that are being used like libraries need to release like
   libraries. Real releases. With appropriate version numbers. So
   that other things that depend on them can express valid dependencies.

2. Extract the relevant code into libraries and release *those*.

3. Things that are not stable enough to be treated as a library
   shouldn't be used that way. Move the things that use the application
   code as library code back into the repo with the thing that they
   are tied to but that we don't want to (or can't) treat like a
   library.

Let's stop making things hard on ourselves and start managing this
code properly.

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Tony Breeds]

On Thu, Mar 15, 2018 at 07:16:11AM +0900, Akihiro Motoki wrote:
> The current version of proposed patches which drops tox_install.sh
> works in our CI. Even if we have neutron>=12.0.0 (queens) or
> horizon>=13.0.0 (queens), if we have "required-projects" in zuul v3
> config, tox-sibling role ensures to install the latest master of
> neutron/horizon. It is okay in our CI.
> 
> On the other hand, this change brings a couple of problems. I think it
> is worth discussed broadly here.
> 
> (1) it makes difficult to run tests in local environment
> We have only released version of neutron/horizon on PyPI. It means
> PyPI version (i.e. queens) is installed when we run tox in our local
> development. Most neutron stadium projects and horizon plugins depends
> on the latest master. Test run in local environment will be broken. We
> need to install the latest neutron/horizon manually. This confuses
> most developers. We need to ensure that tox can run successfully in a
> same manner in our CI and local environments.

This is an issue I agree and one we need to think about but it will be
somewhat mitigated for local development by pbr siblings[1]

In the short term, developers can do something like:

for env in pep8,py35,py27 ; do
   tox -e $env --notest
   .tox/$env/bin/pip install -e /path/to/{horizon,neutron}
   tox -e $env
done

Which is far from ideal but gives as a little breathing room to decide
if we need to revert and try again in a while or persist with the plan
as it stands.

pbr siblings wont fix all the issues we have and still makes consumption of
neutron and horizon (and plugins / stadium projects) difficult outside
of test.
 
> (2) neutron/horizon version in requirements.txt is confusing
> In the cycle-with-milestone model, a new version of neutron/horizon
> will be released only when a release is shipped.
> The code depends on the latest master, but requirements.txt says it
> depends on queens or later. It sounds confusing.

Yes we either need to create a new release-model or switch
neutron/horizon to the cycle-with-intermediary model and encourage
appropriate releases.  I'd really like to avoid publishing daily to pypi.

Yours Tony.

[1]
https://review.openstack.org/#/q/status:open+project:openstack-dev/pbr+branch:master+topic:fix-siblings-entrypoints


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Paul Belanger]

On Wed, Mar 14, 2018 at 10:25:49PM +, Chris Dent wrote:
> On Thu, 15 Mar 2018, Akihiro Motoki wrote:
> 
> > (1) it makes difficult to run tests in local environment
> > We have only released version of neutron/horizon on PyPI. It means
> > PyPI version (i.e. queens) is installed when we run tox in our local
> > development. Most neutron stadium projects and horizon plugins depends
> > on the latest master. Test run in local environment will be broken. We
> > need to install the latest neutron/horizon manually. This confuses
> > most developers. We need to ensure that tox can run successfully in a
> > same manner in our CI and local environments.
> 
> Assuming that ^ is actually the case then:
> 
> This sounds like a really critical issue. We need to be really
> careful about automating the human out of the equation to the point
> where people are submitting broken code just so they can get a good
> test run. That's not great if we'd like to encourage various forms
> of TDD and the like and we also happen to have a limited supply of
> CI resources.
> 
> (Which is not to say that tox-siblings isn't an awesome feature. I
> hadn't really known about it until today and it's a great thing.)
> 
If ansible is our interface for developers to use, it shouldn't be difficult to
reproduce the environments locally to get master. This does mean changing the
developer workflow to use ansible, which I can understand might not be what
people want to do.

The main reason for removing install_tox.sh, is to remove zuul-cloner from our
DIB images, as zuulv3 no longer includes this command. Even running that
locally, would no longer work against git.o.o.

I agree, we should see how to make the migration for local developer
environments better.

Paul

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Chris Dent]

On Thu, 15 Mar 2018, Akihiro Motoki wrote:


(1) it makes difficult to run tests in local environment
We have only released version of neutron/horizon on PyPI. It means
PyPI version (i.e. queens) is installed when we run tox in our local
development. Most neutron stadium projects and horizon plugins depends
on the latest master. Test run in local environment will be broken. We
need to install the latest neutron/horizon manually. This confuses
most developers. We need to ensure that tox can run successfully in a
same manner in our CI and local environments.


Assuming that ^ is actually the case then:

This sounds like a really critical issue. We need to be really
careful about automating the human out of the equation to the point
where people are submitting broken code just so they can get a good
test run. That's not great if we'd like to encourage various forms
of TDD and the like and we also happen to have a limited supply of
CI resources.

(Which is not to say that tox-siblings isn't an awesome feature. I
hadn't really known about it until today and it's a great thing.)

--
Chris Dent   ٩◔̯◔۶   https://anticdent.org/
freenode: cdent tw: @anticdent__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Akihiro Motoki]

The current version of proposed patches which drops tox_install.sh
works in our CI. Even if we have neutron>=12.0.0 (queens) or
horizon>=13.0.0 (queens), if we have "required-projects" in zuul v3
config, tox-sibling role ensures to install the latest master of
neutron/horizon. It is okay in our CI.

On the other hand, this change brings a couple of problems. I think it
is worth discussed broadly here.

(1) it makes difficult to run tests in local environment
We have only released version of neutron/horizon on PyPI. It means
PyPI version (i.e. queens) is installed when we run tox in our local
development. Most neutron stadium projects and horizon plugins depends
on the latest master. Test run in local environment will be broken. We
need to install the latest neutron/horizon manually. This confuses
most developers. We need to ensure that tox can run successfully in a
same manner in our CI and local environments.

(2) neutron/horizon version in requirements.txt is confusing
In the cycle-with-milestone model, a new version of neutron/horizon
will be released only when a release is shipped.
The code depends on the latest master, but requirements.txt says it
depends on queens or later. It sounds confusing.

Thanks,
Akihiro


2018-03-15 6:56 GMT+09:00 Andreas Jaeger :
> On 2018-03-14 20:46, Andreas Jaeger wrote:
>> On 2018-03-14 20:39, Andreas Jaeger wrote:
>>> We now have neutron and horizon in global-requirements and do not need
>>> to install them anymore with tools/tox_install.sh.
>>>
>>> This allows to simplify our jobs and testing.
>>>
>>> Unfortunately, the merging caused now the projects that install neutron
>>> and horizon via tools/tox_install to break with constraints.
>>>
>>> I'm currently pushing up changes for these using topic tox-siblings [1].
>>>
>>> Please merge those - and if you're pushing up changes yourself, let's
>>> use the same topic.
>>>
>>> Sorry for the breakage ;(
>>> Andreas
>>>
>>> [1] Links
>>> https://review.openstack.org/#/q/topic:tox-siblings+(status:open+OR+status:merged)
>>>
>>
>> Note that thanks to the tox-siblings feature, we really continue to
>> install neutron and horizon from git - and not use the versions in the
>> global-requirements constraints file,
>
> Btw. this work is part of what Monty proposed in
> http://lists.openstack.org/pipermail/openstack-dev/2017-November/124676.html
>
> Andreas
> --
>  Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
>   SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>HRB 21284 (AG Nürnberg)
> GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Andreas Jaeger]

On 2018-03-14 20:46, Andreas Jaeger wrote:
> On 2018-03-14 20:39, Andreas Jaeger wrote:
>> We now have neutron and horizon in global-requirements and do not need
>> to install them anymore with tools/tox_install.sh.
>>
>> This allows to simplify our jobs and testing.
>>
>> Unfortunately, the merging caused now the projects that install neutron
>> and horizon via tools/tox_install to break with constraints.
>>
>> I'm currently pushing up changes for these using topic tox-siblings [1].
>>
>> Please merge those - and if you're pushing up changes yourself, let's
>> use the same topic.
>>
>> Sorry for the breakage ;(
>> Andreas
>>
>> [1] Links
>> https://review.openstack.org/#/q/topic:tox-siblings+(status:open+OR+status:merged)
>>
> 
> Note that thanks to the tox-siblings feature, we really continue to
> install neutron and horizon from git - and not use the versions in the
> global-requirements constraints file,

Btw. this work is part of what Monty proposed in
http://lists.openstack.org/pipermail/openstack-dev/2017-November/124676.html

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon][neutron] tools/tox_install changes - breakage with constraints

[Andreas Jaeger]

On 2018-03-14 20:39, Andreas Jaeger wrote:
> We now have neutron and horizon in global-requirements and do not need
> to install them anymore with tools/tox_install.sh.
> 
> This allows to simplify our jobs and testing.
> 
> Unfortunately, the merging caused now the projects that install neutron
> and horizon via tools/tox_install to break with constraints.
> 
> I'm currently pushing up changes for these using topic tox-siblings [1].
> 
> Please merge those - and if you're pushing up changes yourself, let's
> use the same topic.
> 
> Sorry for the breakage ;(
> Andreas
> 
> [1] Links
> https://review.openstack.org/#/q/topic:tox-siblings+(status:open+OR+status:merged)
> 

Note that thanks to the tox-siblings feature, we really continue to
install neutron and horizon from git - and not use the versions in the
global-requirements constraints file,

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev