Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[thomas.morin]

Hi Monty,

Thanks for bringing this up.

Having run into the topic for a few combination of deps, I'll certainly 
agree that we need something better than what we currently have.
I don't feel that I've enough perspective on the whole system and 
practices to give a strong opinion on what we should do though.


A few comments... (below)


On 25/04/2018 16:40, Monty Taylor wrote:


projects with test requirements on git repo urls of other projects
--

There are a bunch of projects that need, for testing purposes, to 
depend on other projects. The majority are either neutron or horizon 
plugins, but conceptually there is nothing neutron or horizon specific 
about the issue. The problem they're trying to deal with is that they 
are a plugin to a service and they need to be able to import code from 
the service they are a plugin to in their unit tests.


(using neutron to avoid being too abstract, but this generalizes to 
other components with plugins)


True, but sometimes a change to a neutron plugin may (with or without a 
need to actually import neutron), need to run against a neutron version 
from git (because the change has a Depends-On a Neutron change, or 
because the change depends on something that is in neutron master but 
not in a release).  We have this when the plugin depends on a new or 
fixed behavior.


While this case can in theory be fixed by moving the code introducing 
the fixed or new behavior into neutron-lib,  it doesn't mean that this 
is always feasible (because the work required to move this part of the 
code to neutron-lib hasn't happened).







unwinding things


There are a few different options, but it's important to keep in mind 
that we ultimately want all of the following:


* The code works
* Tests can run properly in CI
* "Depends-On" works in CI so that you can test changes cross-repo


Note that this was true with tools/tox_install.sh, but broke when it was 
removed for a job such as legacy-networking-bgpvpn-dsvm-functional (see 
[1]) which does not inherit from zuul tox jobs, but still relies 
ultimately on zuul to run the test.


[1] 
http://logs.openstack.org/41/558741/11/check/legacy-networking-bgpvpn-dsvm-functional/86a743c/



* Tests can run properly locally for developers


(Broke when tools/tox_install.sh was abandoned, currently causing minor 
pain to lots of people working on neutron-plugins unless py27-dev hacks 
are in place in their project)




* Deployment requirements are accurately communicated to deployers


Was definitely improved by removing tools/tox_install.sh!




Specific Suggestions


As there are a few different scenarios, I want to suggest we do a few 
different things.


* Prefer interface libraries on PyPI that projects depend on

Like python-openstackclient and osc-lib, this is the *best* approach
for projects with plugins. Such interface libraries need to be able to 
do intermediate releases - and those intermediate releases need to not 
break the released version of the projects. This is the hardest and 
longest thing to do as well, so it's most likely to be a multi-cycle 
effort.


I would object to "best", for the following reasons:
- because this is not the starting point, the effort to librarize code 
is significant, and it's seems a fact that people don't rush to do it
- there is a practical drawback of doing that: even for project that 
have compatible release cycle, we have overhead of having to release 
e.g. neutron-lib with the change before we can consume it in neutron or 
a neutron plugin (and we have overhead to test the changes as well, with 
extra jobs to test against master or local .zuul.yaml hacks to force 
Depends-On to test what we want e.g. [x] ) ; a situation that would 
avoid this overhead would I think be superior


[x] https://review.openstack.org/#/c/557660/



* Treat inter-plugin depends as normal library depends

If networking-bgpvpn depends on networking-bagpipe and networking-odl, 
then networking-bagpipe and networking-odl need to be released to PyPI 
just like any other library in OpenStack. These are real runtime 
dependencies.


Juste a side note here: networking-bagpipe and networking-odl provide 
components drivers for their corresponding drivers in networking-bgpvpn, 
they aren't strict runtime dependencies, but only dependencies for a 
scenario where their driver is used. Given that, they were moved as 
test-requirements dependencies recently (only required to run unit tests).


The situation for these drivers is a bit in flux though:
- ODL: the bgpvpn driver for ODL is a v1 driver that became obsolete, 
there is a v2 driver sitting entirely in networking-odl
- bagpipe: the bgpvpn driver for bagpipe could be moved to 
networking-bagpipe entirely  -- the one reason why it hasn't happened 
(apart from inertia) is that is it the reference driver for the 
networking-bgpvpn project, and removing it from 

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[thomas.morin]

On 25/04/2018 18:40, Jeremy Stanley wrote:

This came up again a few days ago for sahara-dashboard. We talked
through some obvious alternatives to keep its master branch from
depending on an unreleased state of horizon and the situation today
is that plugin developers have been relying on developing their
releases in parallel with the services. Not merging an entire
development cycle's worth of work until release day (whether that's
by way of a feature branch or by just continually rebasing and
stacking in Gerrit) would be a very painful workflow for them, and
having to wait a full release cycle before they could start
integrating support for new features in the service would be equally
unfortunate.


+1



As for merging the plugin and service repositories, they tend to be
developed by completely disparate teams so that could require a fair
amount of political work to solve. Extracting the plugin interface
into a separate library which releases more frequently than the
service does indeed sound like the sanest option, but will also
probably take quite a while for some teams to achieve (I gather
neutron-lib is getting there, but I haven't heard about any work
toward that end in Horizon yet).


+1

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[Boden Russell]

On 4/25/18 10:13 PM, Shu M. wrote:
> Hi folks,
> 
>> unwinding things
>> 
>>
>> There are a few different options, but it's important to keep in mind
>> that we ultimately want all of the following:
>>
>> * The code works
>> * Tests can run properly in CI
>> * "Depends-On" works in CI so that you can test changes cross-repo
>> * Tests can run properly locally for developers
>> * Deployment requirements are accurately communicated to deployers
> 
> One more thing:
> * Test environments between CI and local should be as same as possible.
> 
> To run tests in CI and local successfully, I have tried to add new
> testenv for local into tox.ini
> (https://review.openstack.org/#/c/564099/4/tox.ini
> ) as alternative
> solution last night, this would be same as adding new requirements.txt
> for local check. This seems to run fine, but this might make difference
> in environments between CI and local. So I can not conclude this is the
> best way for now.

I'd like to echo this point from a neutron plugin (project) perspective.

While we can move all our inter-project dependencies to requirements now
[1], this does not address how we can run tox targets or devstack
locally with master branches (to test changes locally before submitting
to gate).

To mitigate running tox locally we've introduced new targets that
manually install the inter-project dependencies [2] in editable mode.
While this works, IMHO it's not optimal and furthermore requires
"special" steps if you want to add some changes to those editable
projects and run with them.

And we've done something similar for devstack [3].

Finally, we also have some periodic jobs used to pre-validate our shared
neutron-lib project using master branches as defined by the
periodic-jobs-with-neutron-lib-master template. Certainly we want to
keep these working.


Frankly it's been a bit of a cat-and-mouse game to keep up with the
infra/zuul changes in the past 2 releases so it's possible what we've
done could be improved upon. If that's the case please do let me know so
we can works towards and optimized approach.

Thanks


[1] https://review.openstack.org/#/c/554292/
[2] https://review.openstack.org/#/c/555005/5/tox.ini
[3] https://review.openstack.org/#/c/555005/5/devstack/lib/nsx_common

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[William M Edmonds]

Monty Taylor  wrote on 04/25/2018 09:40:47 AM:
...
> Introduce a whitelist of git repo urls, starting with:
>
>* https://git.openstack.org/openstack/neutron
>* https://git.openstack.org/openstack/horizon
>
We would also need to include at least nova (e.g. [1]) and ceilometer (e.g.
[2]).

[1] https://github.com/openstack/nova-powervm
[2] https://github.com/openstack/ceilometer-powervm
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[Shu M.]

Hi folks,

> unwinding things
> 
>
> There are a few different options, but it's important to keep in mind
> that we ultimately want all of the following:
>
> * The code works
> * Tests can run properly in CI
> * "Depends-On" works in CI so that you can test changes cross-repo
> * Tests can run properly locally for developers
> * Deployment requirements are accurately communicated to deployers

One more thing:
* Test environments between CI and local should be as same as possible.

To run tests in CI and local successfully, I have tried to add new testenv
for local into tox.ini (https://review.openstack.org/#/c/564099/4/tox.ini)
as alternative solution last night, this would be same as adding new
requirements.txt for local check. This seems to run fine, but this might
make difference in environments between CI and local. So I can not conclude
this is the best way for now.

>From view of horizon plugin developer, the one of issue on horizon and
plugins is the feature gap due to unsufficient communication.
Merging developing repository may give good effects for this issue, if
horizon can separate each panels into plugins.

Thanks,
Shu Muto
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[Ivan Kolodyazhny]

Hi team,

I'm speaking mostly from Horizon side, but it should be pretty the same for
others.

We started a discussion today at the Horizon's meeting but we don't have
any decision now.

For the current release cycle, it looks reasonable to test plugins over the
latest master on gates. We're thinking to introduce horizon-lib but we need
further discussions on it.

Horizon follows stable policy and we try to do our best to not break any
existing plugin. Unfortunately, due to some cross-projects
miscommunications, there were some issues with plugins this cycle. I'm
ready to work with plugins team to fix these issues asap.

To prevent such issues in the future, I think it would be good to have
cross-project jobs on Horizon's gates too. We should run at least plugins
unit-tests against Horizon's proposed patch to make sure that we don't
break anything in plugins. E.g. if we drop some deprecated feature or make
an incompatible change, such job will notify us that we need to update
a plugin first before merging patch to Horizon. I don't know what is the
best way to implement such jobs so it would be good to get some inputs and
help from Infra team here.



Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Wed, Apr 25, 2018 at 7:40 PM, Jeremy Stanley  wrote:

> On 2018-04-25 12:03:54 -0400 (-0400), Doug Hellmann wrote:
> [...]
> > Especially now with lower-constraints jobs in place, having plugins
> > rely on features only available in unreleased versions of service
> > projects doesn't make a lot of sense. We test that way *between*
> > services using integration tests that use the REST APIs, but we
> > also have some pretty strong stability requirements in place for
> > those APIs.
> [...]
>
> This came up again a few days ago for sahara-dashboard. We talked
> through some obvious alternatives to keep its master branch from
> depending on an unreleased state of horizon and the situation today
> is that plugin developers have been relying on developing their
> releases in parallel with the services. Not merging an entire
> development cycle's worth of work until release day (whether that's
> by way of a feature branch or by just continually rebasing and
> stacking in Gerrit) would be a very painful workflow for them, and
> having to wait a full release cycle before they could start
> integrating support for new features in the service would be equally
> unfortunate.
>
> As for merging the plugin and service repositories, they tend to be
> developed by completely disparate teams so that could require a fair
> amount of political work to solve. Extracting the plugin interface
> into a separate library which releases more frequently than the
> service does indeed sound like the sanest option, but will also
> probably take quite a while for some teams to achieve (I gather
> neutron-lib is getting there, but I haven't heard about any work
> toward that end in Horizon yet).
> --
> Jeremy Stanley
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[Jeremy Stanley]

On 2018-04-25 12:03:54 -0400 (-0400), Doug Hellmann wrote:
[...]
> Especially now with lower-constraints jobs in place, having plugins
> rely on features only available in unreleased versions of service
> projects doesn't make a lot of sense. We test that way *between*
> services using integration tests that use the REST APIs, but we
> also have some pretty strong stability requirements in place for
> those APIs.
[...]

This came up again a few days ago for sahara-dashboard. We talked
through some obvious alternatives to keep its master branch from
depending on an unreleased state of horizon and the situation today
is that plugin developers have been relying on developing their
releases in parallel with the services. Not merging an entire
development cycle's worth of work until release day (whether that's
by way of a feature branch or by just continually rebasing and
stacking in Gerrit) would be a very painful workflow for them, and
having to wait a full release cycle before they could start
integrating support for new features in the service would be equally
unfortunate.

As for merging the plugin and service repositories, they tend to be
developed by completely disparate teams so that could require a fair
amount of political work to solve. Extracting the plugin interface
into a separate library which releases more frequently than the
service does indeed sound like the sanest option, but will also
probably take quite a while for some teams to achieve (I gather
neutron-lib is getting there, but I haven't heard about any work
toward that end in Horizon yet).
-- 
Jeremy Stanley


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[Doug Hellmann]

Excerpts from Monty Taylor's message of 2018-04-25 16:40:47 +0200:
> Hi everybody,
> 
> We've been working on navigating through from an interesting situation 
> over the past few months, but there isn't a top-level overview of what's 
> going on with it. That's my bad - I've been telling AJaeger I was going 
> to send an email out for a while.
> 
> projects with test requirements on git repo urls of other projects
> --
> 
> There are a bunch of projects that need, for testing purposes, to depend 
> on other projects. The majority are either neutron or horizon plugins, 
> but conceptually there is nothing neutron or horizon specific about the 
> issue. The problem they're trying to deal with is that they are a plugin 
> to a service and they need to be able to import code from the service 
> they are a plugin to in their unit tests.
> 
> To make things even more complicated, some of the plugins actually 
> duepend on each other for real, not just as a "we need this for testing"
> 
> There is trouble in paradise though - which is that we don't allow git 
> urls in requirements files. To work around this, the projects in 
> question added additional pip install lines to a tox_install.sh script - 
> essentially bypassing the global-requirements process and system 
> completely.
> 
> This went unnoticed in a general sense until we started working through 
> removing the use of zuul-cloner which is not needed any longer in Zuul v3.
> 
> unwinding things
> 
> 
> There are a few different options, but it's important to keep in mind 
> that we ultimately want all of the following:
> 
> * The code works
> * Tests can run properly in CI
> * "Depends-On" works in CI so that you can test changes cross-repo
> * Tests can run properly locally for developers
> * Deployment requirements are accurately communicated to deployers
> 
> The approach so far
> ---
> 
> The approach so far has been releasing service projects to PyPI and 
> reworking the projects to depend on those releases.
> 
> This approach takes advantage of the tox-siblings feature in the gate to 
> ensure we're cross-testing master of projects with each other.
> 
> tox-siblings
> ---
> 
> There is a feature in the Zuul tox jobs we refer to as "tox-siblings" 
> (this is because historically - wow we have historical context for zuul 
> v3 now - it was implemented as a separate role) What it does is ensure 
> that if you are running a tox job and you add additional projects to 
> required-projects in the job config, that the git versions of those 
> projects will be installed into the tox virtualenv - but only for 
> projects that would have been installed by tox otherwise. This way 
> required-projects is both safe to use and has the effect you'd expect.
> 
> tox-siblings is intended to enable ADDITIONALLY cross-testing projects 
> that otherwise have a normal dependency relationship in the gate. People 
> have been adding jobs like cross-something-something or something-tips 
> in an ad-hoc manner for a while - and in many cases the git parts of 
> that were actually somewhat not correct - so this is an attempt to 
> provide the thing people want in those scenarios in a consistent manner. 
> But it always should be helper logic for more complex gate jobs, not as 
> a de-facto part of a project's basic install.
> 
> Current Approach is wrong
> 
> 
> Unfortunately, as part of trying to unwind the plugins situation, we've 
> walked ourselves into a situation where the gate is the only thing that 
> has the correct installation information for some projects, and that's 
> not good.
> 
>  From a networking plugin approach the "depend on release and use 
> tox-siblings" assumes that 'depend on release of neutron' is or can be 
> the common case with the ability to add a second tox job to check master 
> against master.
> 
> If that's not a real thing, then depending on releases + tox_siblings in 
> the gate is solving the wrong problem.
> 
> Specific Suggestions
> 
> 
> As there are a few different scenarios, I want to suggest we do a few 
> different things.
> 
> * Prefer interface libraries on PyPI that projects depend on
> 
> Like python-openstackclient and osc-lib, this is the *best* approach
> for projects with plugins. Such interface libraries need to be able to 
> do intermediate releases - and those intermediate releases need to not 
> break the released version of the projects. This is the hardest and 
> longest thing to do as well, so it's most likely to be a multi-cycle effort.
> 
> * Treat inter-plugin depends as normal library depends
> 
> If networking-bgpvpn depends on networking-bagpipe and networking-odl, 
> then networking-bagpipe and networking-odl need to be released to PyPI 
> just like any other library in OpenStack. These are real runtime 
> dependencies.
> 
> Yes, this is more coordination work, 

Re: [openstack-dev] [requirements][horizon][neutron] plugins depending on services

[Jeremy Stanley]

On 2018-04-25 16:40:47 +0200 (+0200), Monty Taylor wrote:
[...]
> * Relax our rules about git repos in test-requirements.txt
> 
> Introduce a whitelist of git repo urls, starting with:
> 
>   * https://git.openstack.org/openstack/neutron
>   * https://git.openstack.org/openstack/horizon
> 
> For the service projects that have plugins that need to test against the
> service they're intending to be used with in a real installation. For those
> plugin projects, actually put the git urls into test-requirements.txt. This
> will make the gate work AND local development work for the scenarios where
> the thing that is actually needed is always testing against tip of a
> corresponding service.
[...]

If this is limited to test-requirements.txt and doesn't spill over
into requirements.txt then it _might_ be okay, though it still seems
like we'd need some sort of transition around release stable
branching to indicate what URLs should be used to correspond to
those branches. We've been doing basically that with release-time
edits to per-repo tox-install.sh scripts already, so maybe the same
workflow can be reused on test-requirements.txt a well?
-- 
Jeremy Stanley


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev