Re: [openstack-dev] [neutron][stadium][networking] Seeking proposals for non-voting Stadium projects in Neutron check queue

[Michel Peterson]

On Sun, Sep 30, 2018 at 3:43 AM Miguel Lavalle  wrote:

> The next step is for each project to propose the jobs that they want to
> run against Neutron patches.
>

This is fantastic. Do you plan to have all patches under a single topic for
easier tracking? I'll be handling the proposal of these jobs for
networking-odl and would like to know this before sending them for review.

In addition, since these will be non-voting for Stadium projects, how will
the mechanics be to avoid breakage of such projects?
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [doc][i18n][infra][tc] plan for PDF and translation builds for documentation

[Michel Peterson]

On Thu, Sep 13, 2018 at 1:09 AM, Doug Hellmann 
wrote:

> The longer version is that we want to continue to use the existing
> tox environment in each project as the basis for the job, since
> that allows teams to control the version of python used, the
> dependencies installed, and add custom steps to their build (such
> as for pre-processing the documentation). So, the new or updated
> job will start by running "tox -e docs" as it does today. Then it
> will run Sphinx again with the instructions to build PDF output,
> and copy the results into the directory that the publish job will
> use to sync to the web server. And then it will run the scripts to
> build translated versions of the documentation as HTML, and copy
> the results into place for publishing.
>

Just a question out of curiosity. You mention that we still want to use the
docs environment because it allows fine grained control over how the
documentation is created. However, as I understand, the PDF output will
happen in a more standardized way and outside of that fine grained control,
right? That couldn't lead to differences in both documentations? Do we have
to even worry about that?
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [networking-odl][networking-bgpvpn][ceilometer] all requirement updates are currently blocked

[Michel Peterson]

On Tue, Sep 11, 2018 at 7:29 AM, Tony Breeds 
wrote:

>
> So I think we have the required reviews lined up to fix master, but they
> need votes from zuul and core teams.
>
>
Thanks a lot for the work, Tony. On the n-odl side, when the Depends-On
gets merged I'll give it a +W.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [networking-odl][networking-bgpvpn][ceilometer] all requirement updates are currently blocked

[Michel Peterson]

On Wed, Sep 5, 2018 at 6:03 PM, Matthew Thode 
wrote:

> On 18-08-31 19:52:09, Matthew Thode wrote:
> > The requirements project has a co-installability test for the various
> > projects, networking-odl being included.
> >
> > Because of the way the dependancy on ceilometer is done it is blocking
> > all reviews and updates to the requirements project.
> >
> > http://logs.openstack.org/96/594496/2/check/requirements-
> integration/8378cd8/job-output.txt.gz#_2018-08-31_22_54_49_357505
>
> The requirements team has gone ahead and made a aweful hack to get gate
> unwedged.  The commit message is a very good summary of our reasoning
> why it has to be this way for now.  My comment explains our plan going
> forward (there will be a revert prepared as soon as this merges for
> instance).
>
> step 1. merge this
> step 2. look into and possibly fix our tooling (why was the gitref
> addition not rejected by gate)
> step 3. fix networking-odl (release ceilometer)
> step 4. unmerge this
>

I remember that before landing the problematic patch [1] there was some
discussion around it. Basically the problem was not n-odl but ceilometer
not being in pypi, but we never foresaw this problem.

Now that the problem is so critical, the question is how can we, from the
n-odl team, help in fixing this? I am open to help in any effort that
involves n-odl or any other project.

Sorry this message fell through the cracks and I didn't answer before.

PS: I'm CCing Mike Kolesnik to this email, as he will be going to the PTG
and can represent n-odl.

[1] https://review.openstack.org/557370/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] tox-siblings alternative for local testing

[Michel Peterson]

On Wed, Aug 29, 2018 at 9:06 AM, Takashi Yamamoto 
wrote:

> is there any preferred solution for this?
> i guess the simplest solution is to make an intermediate release of neutron
> and publish it on pypi.  i wonder if it's acceptable or not.
>

There are pre releases available in PyPI [1]. You can use those from your
requirements like we did in n-odl [2].

That might be an acceptable solution.

[1] https://pypi.org/project/neutron/#history
[2] https://review.openstack.org/#/c/584791/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][python3] Neutron and stadium - python 3 community goal changes coming soon

[Michel Peterson]

On Thu, Aug 30, 2018 at 12:14 AM, Nate Johnston 
wrote:

> Progress is also being tracked in a wiki page [4].
>
> [4] https://wiki.openstack.org/wiki/Python3
>

That wiki page should only track teams or subteams should also be included
there? I'm asking because it seems that some subteams appear there while
the majority doesn't and perhaps we want to standarize that.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goal][python3] week 3 update

[Michel Peterson]

On Thu, Aug 30, 2018 at 3:11 AM, Doug Hellmann 
wrote:

> | import zuul job settings from project-config | openstack/networking-odl
>  | https://review.openstack.org/597870 | master|
> | switch documentation job to new PTI  | openstack/networking-odl
>  | https://review.openstack.org/597871 | master|
> | add python 3.5 unit test job | openstack/networking-odl
>  | https://review.openstack.org/597872 | master|
> | add python 3.6 unit test job | openstack/networking-odl
>  | https://review.openstack.org/597873 | master|
> | import zuul job settings from project-config | openstack/networking-odl
>  | https://review.openstack.org/597911 | stable/ocata  |
> | import zuul job settings from project-config | openstack/networking-odl
>  | https://review.openstack.org/597923 | stable/pike   |
> | import zuul job settings from project-config | openstack/networking-odl
>  | https://review.openstack.org/597938 | stable/queens |
> | import zuul job settings from project-config | openstack/networking-odl
>  | https://review.openstack.org/597953 | stable/rocky  |


In the case of networking-odl I know we also need a 'fix tox python3
overrides' patch but that was not generated. I'm wondering if I should do
it manually or it will be generated at a later stage?
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goal][python3] week 3 update

[Michel Peterson]

On Thu, Aug 30, 2018 at 3:11 AM, Doug Hellmann 
wrote:

> > OK, there are somewhere just over 100 patches for all of the neutron
> > repositories, so I'm going to wait for a quieter time of day to submit
> > them to avoid blocking other, smaller, bits of work.
> >
> > Doug
>
> Those patches are up for review now. - Doug
>
>
Doug, just a heads up the tool for python3-first is duplicating the same
block of code (e.g. https://review.openstack.org/#
/c/597873/1/.zuul.d/project.yaml ) and in some cases duplicating code that
already exists (e.g. https://review.openstack.org/#
/c/597872/1/.zuul.d/project.yaml ).

Perhaps it will be good to review the tool before moving forward.

Best,
M

P.S. I've sent you the same through #openstack-infra
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goal][python3] week 3 update

[Michel Peterson]

On Mon, Aug 27, 2018 at 10:37 PM, Doug Hellmann 
wrote:

>
> If your team is ready to have your zuul settings migrated, please
> let us know by following up to this email. We will start with the
> volunteers, and then work our way through the other teams.
>

The networking-odl team is willing to volunteer for this.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra] Greasemonkey script to see CI progress on OpenStack's Gerrit

[Michel Peterson]

Hello everyone,

I've created a greasemonkey script that will show you the status of
the current CI run on the OpenStack Gerrit in real time. I put it
together really quickly so there is room for improvement.

You can find it here:
https://gist.github.com/mpeterson/bb351543c4abcca8e7bb1205fcea4c75

I am wondering if this would be an interesting thing for you guys to
add to Gerrit directly? I think it's very useful to have.

Should I propose a patch to include it in review.o.o ?

Best,
M

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][requirements] uncapping eventlet

[Michel Peterson]

On Fri, Apr 20, 2018 at 11:06 PM, Doug Hellmann 
wrote:

>
> We have made great progress on this but we do still have quite a
> few of these patches that have not been approved.  Many are failing
> test jobs and will need a little attention ( the failing requirements
> jobs are real problems and rechecking will not fix them).  If you
> have time to help, please jump in and take over a patch and get it
> working.
>
> https://review.openstack.org/#/q/status:open+topic:uncap-eventlet
>
>
I did a script to fix those and I've submitted patches.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][neutron][requirements][pbr]Use git+https line in requirements.txt break the pip install

[Michel Peterson]

On Wed, Apr 18, 2018 at 12:02 PM, Michel Peterson <mic...@redhat.com> wrote:

> How can we fix this? There are several ways I can think of the top of my
> head:
>
>
>1. When encountered with edge cases like this one, first install that
>dependency with a manual pip run [2]
>2. Modify pbr to handle these situations by handling the installation
>of those depenencies differently with a workaround to the current
>functionality of pip
>3. Leverage on the work of corvus [3] to not only do what that patch
>is doing, but also including the checked out path of the dependency in
>PIP_FIND_LINKS, that way pip knows how to solve the issue.
>
> All these solutions have different set of pros and cons, but I favor #3 as
> the long term solution, #1 as short term and I think #2 requires further
> analysis by the pbr team.
>

I forgot to add the reference on where to add the PIP_FIND_LINKS for
solution #3, here you go:

https://github.com/openstack-dev/devstack/blob/f99d1771ba1882dfbb69186212a197edae3ef02c/inc/python#L362
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][neutron][requirements][pbr]Use git+https line in requirements.txt break the pip install

[Michel Peterson]

Hi, I'm one of the networking-odl core devs.

On Wed, Apr 18, 2018 at 5:48 AM, Jeffrey Zhang 
wrote:

>
> Recently, one of networking-odl package breaks kolla's gate[0]. The direct
> issue is ceilometer is added in networking-odl's requirements.txt file[1]
>

This is an issue that concerns me too. First off let me start with a simple
solution, which is to install ceilometer from git before requiring
networking-odl. Also, if networking-odl is installed through devstack's
enable_plugin this issue wouldn't arise (as the plugin.sh takes care of
installing ceilometer before installing networking-odl).

Still, I see this as a problem, I just didn't find a way to solve it in
general, except ceilometer being published to PyPI. What happened then is I
got caught up in other priorities that took bandwidth away from it and
kinda forgot about it.


>
> Then when install network-odl with upper-contraints.txt file, it will
> raise error like
>
> $ pip install -c https://git.openstack.org/cgit/openstack/requirements/
> plain/upper-constraints.txt ./networking-odl
> ...
> collecting networking-bgpvpn>=8.0.0 (from networking-odl==12.0.1.dev54)
>   Downloading http://pypi.doubanio.com/packages/5a/e5/
> 995be0d53d472f739a7a0bb6c9d9fecbc4936148651aaf56d39f3b65b1f1
> /networking_bgpvpn-8.0.0-py2-none-any.whl (172kB)
> 100% || 174kB 12.0MB/s
> Collecting ceilometer (from networking-odl==12.0.1.dev54)
>   Could not find a version that satisfies the requirement ceilometer (from
> networking-odl==12.0.1.dev54) (from versions: )
> No matching distribution found for ceilometer (from
> networking-odl==12.0.1.dev54)
>
>
> But if you just install the networking-odl's requirements.txt file, it
> works
>
>
> $ pip install -c https://git.openstack.org/cgit/openstack/requirements/
> plain/upper-constraints.txt -r ./networking-odl/requirements.txt
> ...
> Obtaining ceilometer from git+https://git.openstack.org/
> openstack/ceilometer@master#egg=ceilometer (from -r
> networking-odl/requirements.txt (line 21))
>   Cloning https://git.openstack.org/openstack/ceilometer (to revision
> master) to /home/jeffrey/.dotfiles/virtualenvs/test/src/ceilometer
> ...
>
>
> Is this expected? and how could we fix this?
>

This is an interesting case of how pip works differently when installing
from a requirements file or from a folder (as it would happen with -e or
the first command you issued). While in the former it knows how to solve
the dependencies correctly, in the second it actually relies in the
setup.py file to install. That means it goes into pbr's realm and does not
use the requirements at all. So let's analyse what happens in pbr.

Internally in PBR what is doing is reading the requirements.txt, finding
the -e line, reading it's comment that says #egg=ceilometer and adding that
as a requirement [1]. What is failing to do though, is to instruct pip to
fetch it from the git repository (as the requirements file would do).
Sadly, this is not only a problem of pbr but it's also a limitation of the
current state of pip and the corresponding PEPs, which apparently is
already addressed for the long term with new PEPs and upcoming changes to
pip.

How can we fix this? There are several ways I can think of the top of my
head:


   1. When encountered with edge cases like this one, first install that
   dependency with a manual pip run [2]
   2. Modify pbr to handle these situations by handling the installation of
   those depenencies differently with a workaround to the current
   functionality of pip
   3. Leverage on the work of corvus [3] to not only do what that patch is
   doing, but also including the checked out path of the dependency in
   PIP_FIND_LINKS, that way pip knows how to solve the issue.

All these solutions have different set of pros and cons, but I favor #3 as
the long term solution, #1 as short term and I think #2 requires further
analysis by the pbr team.


Hope my contribution helped to clarify this issue.


[1]:
https://github.com/openstack-dev/pbr/blob/7767c44ab1289ed7d1cc4f9e12986bef07865d5c/pbr/packaging.py#L168

[2]:
https://github.com/openstack/networking-odl/blob/aa3acb23a5736f128fee0a514a588b9035551d88/devstack/entry_points#L259

[3]: https://review.openstack.org/549252/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] Generalized issues in the unit testing of ML2 mechanism drivers

[Michel Peterson]

Through my work in networking-odl I've found what I believe is an issue
present in a majority of ML2 drivers. An issue I think needs awareness so
each project can decide a course of action.

The issue stems from the adopted practice of importing
`neutron.tests.unit.plugins.ml2.test_plugin` and creating classes with noop
operation to "inherit" tests for free [1]. The idea behind is nice, you
inherit >600 tests that cover several scenarios.

There are several issues of adopting this pattern, two of which are
paramount:

1. If the mechanism driver is not loaded correctly [2], the tests then
don't test the mechanism driver but still succeed and therefore there is no
indication that there is something wrong with the code. In the case of
networking-odl it wasn't discovered until last week, which means that for
>1 year it this was adding PASSed tests uselessly.

2. It gives a false sense of reassurance. If the code of those tests is
analyzed it's possible to see that the code itself is mostly centered
around testing the REST endpoint of neutron than actually testing that the
mechanism succeeds on the operation it was supposed to test. As a result of
this, there is marginally added value on having those tests. To be clear,
the hooks for the respective operations are called on the mechanism driver,
but the result of the operation is not asserted.

I would love to hear more voices around this, so feel free to comment.

Regarding networking-odl the solution I propose is the following:
  **First**, discard completely the change mentioned in the footnote #2.
  **Second**, create a patch that completely removes the tests that follow
this pattern.
  **Third**, incorporate the neutron tempest plugin into the CI and rely on
that for assuring coverage of the different scenarios.

Also to mention that when discovered this issue in networking-odl we took a
decision not to merge more patches until the PS of footnote #2 was
addressed. I think we can now decide to overrule that decision and proceed
as usual.



[1]: http://codesearch.openstack.org/?q=class%20.*\(.*TestMl2
[2]: something that was happening in networking-odl and addressed by
https://review.openstack.org/#/c/523934
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron][oslo.db] db retry madness

[Michel Peterson]

Thanks for this most interesting and informative exchange.

On Fri, Sep 8, 2017 at 5:20 PM, Michael Bayer  wrote:
> On Fri, Sep 8, 2017 at 8:53 AM, Kevin Benton  wrote:
> > Since the goal of that patch is to deal with deadlocks, the retry can't
> > happen down at that level, it will need to be somewhere further up the call
> > stack since the deadlock will put the session in a rollback state.
>
>
> OK when I was talking to Michel about this, I first mentioned that to
> get retry within this in-a-transaction block would require adding a
> savepoint to the mix, but he seemed to indicate that the method can
> also run by itself, but since there's no session.begin() there then
> that matches with my initial impression that the retry has to be at
> the level which the transaction starts, and that's not here.


I think there is also a deal of madness on how things are implemented
in this particular project. I'll review the way it's handled and
adjust accordingly, and then implement the right retries in the
correct places. I think my strategy will be to first start with using
a decorator based on wrap_db_retry to fix the bug. After that I'll
start, in another set of patches, the migration from the "legacy"
pattern to the "new" enginefacade. What do you think? Or given that
enginefacade already diminishes the possibilities of deadlocks I
should already start with the migration to enginefacade and then with
that already in place

> > For the functions being called outside of an active session, they should
> > switch to accepting a context to handle the enginefacade switch.

ACK.

> >>  3a.  Is this a temporary measure to work around legacy patterns?
> >
> > Yes. When that went in we had very little adoption of the enginefacade. Even
> > now we haven't fully completed the transition so checking for is_active
> > covers the old code paths.
>
> OK, so this is where they are still calling session.begin()
> themselves.Are all these session.begin() calls in projects under
> the "openstack" umbrella so I can use codesearch ?

So, to make sure I understand correctly. With enginefacade instead of
calling session.begin() to handle the transactions, when using
'reader' o 'writer' we are already inside a transaction, right? If
that's correct, no matter if decorating or using a context manager, in
both cases the transaction lives for the context of the decorated
function or the with-stmt context, right?

> > As we continue migrating to the enginefacade, the cases where you can get a
> > reference to a session that actually has 'is_active==False' are going to
> > keep dwindling. Once we complete the switch and remove the legacy session
> > constructor, the decorator will just be adjusted to check if there is a
> > session attached to the context to determine if retries are safe since
> > is_active will always be True.

Therefore, on implementation of enginefacade retry_if_session_inactive
is rendered useless and because of the deepcopy retry_db_errors in our
usecase is already useless. As a result we are better off with using
wrap_db_retry to create a decorator parametrised according to our
needs (ie: max retries, exception checker, etc), right? And for
exception checker I should use the is_retriable from neutron-lib
instead of the neutron db api, right? (it's a shame that MAX_RETRIES
is not also on neutron-lib and only in neutron)

Thanks to you both.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev