Re: [openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

2016-06-10 Thread Robert Collins
The underlying problem is as you say that distutils and setuptools and
wheel are all different, and until there is a PEP addressing this
*and* implemented in both wheel and setuptools, we won't get
consistent behaviour. I'm pretty sure Daniel Hoth is interested in
making this happen - there have been threads about it on distutils-sig
inside the last year, but we (me/sachi/steven) hadn't gotten onto it
as a thing to help with. I suggest chatting to Daniel as a first point
of call to see where his effort is at and what help is needed.

Note that we *need* the wheel and setuptools behaviour to be
consistent, or we will have differences between distro package
building, devstack, and ansible uses.. amongst others.

-Rob

On 11 June 2016 at 13:50, Morgan Fainberg  wrote:
> There has been a bit of back[1] and forth[2][3][4][5] between at least one
> packaging group and a few folks who are trying to define data files (config)
> in the setup.cfg to aid/ease installation within virtual environments.
>
> From what I can tell, there has been an issue with setuptools that makes
> this a particularly sticky situation and difficult to address in PBR via an
> option like --sysconfdir due to a disagreement between setuptools and
> disttools disagreeing upon the meaning of "data-files".  [6]
>
> Before this turns into a nightmare of add-patches/revert/add/revert and
> making things highly inconsistent within OpenStack, I'd like to get feedback
> from the packaging teams on where this impacts them. I know that a number of
> folks carry effectively this patch internally to make working with VENVs
> easier.
>
> We should absolutely address this in setuptools/distutils/etc but a clear
> direction forward so that the projects under OpenStack remain consistent for
> users, developers, and packagers would be good at this point.
>
> I know that keystone has had a lot of work done to ensure we can work in
> most "general" environments, but installing the data-files within a VENV is
> much more simple than a bunch of special casing to "find" the config files.
>
> In short, I think OpenStack needs to define (even if it's a short period of
> time) what we consider "data-files", we can always revisit this when/if we
> have a clear path forward via PBR, setuptools, disttools, etc.
>
> [1] https://review.openstack.org/#/c/322086/
> [2] https://review.openstack.org/#/c/326152/
> [3] http://git.openstack.org/cgit/openstack/neutron/tree/setup.cfg#n24
> [4] http://git.openstack.org/cgit/openstack/gnocchi/tree/setup.cfg#n87
> [5] http://git.openstack.org/cgit/openstack/aodh/tree/setup.cfg#n30
> [6] https://review.openstack.org/#/c/274077/
>
> Cheers,
> --Morgan
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [tc][pbr][packaging][all] Definition of Data Files (config) in setup.cfg

2016-06-10 Thread Morgan Fainberg
There has been a bit of back[1] and forth[2][3][4][5] between at least one
packaging group and a few folks who are trying to define data files
(config) in the setup.cfg to aid/ease installation within virtual
environments.

>From what I can tell, there has been an issue with setuptools that makes
this a particularly sticky situation and difficult to address in PBR via an
option like --sysconfdir due to a disagreement between setuptools and
disttools disagreeing upon the meaning of "data-files".  [6]

Before this turns into a nightmare of add-patches/revert/add/revert and
making things highly inconsistent within OpenStack, I'd like to get
feedback from the packaging teams on where this impacts them. I know that a
number of folks carry effectively this patch internally to make working
with VENVs easier.

We should absolutely address this in setuptools/distutils/etc but a clear
direction forward so that the projects under OpenStack remain consistent
for users, developers, and packagers would be good at this point.

I know that keystone has had a lot of work done to ensure we can work in
most "general" environments, but installing the data-files within a VENV is
much more simple than a bunch of special casing to "find" the config files.

In short, I think OpenStack needs to define (even if it's a short period of
time) what we consider "data-files", we can always revisit this when/if we
have a clear path forward via PBR, setuptools, disttools, etc.

[1] https://review.openstack.org/#/c/322086/
[2] https://review.openstack.org/#/c/326152/
[3] http://git.openstack.org/cgit/openstack/neutron/tree/setup.cfg#n24
[4] http://git.openstack.org/cgit/openstack/gnocchi/tree/setup.cfg#n87
[5] http://git.openstack.org/cgit/openstack/aodh/tree/setup.cfg#n30
[6] https://review.openstack.org/#/c/274077/

Cheers,
--Morgan
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

2016-06-10 Thread Morgan Fainberg
On Fri, Jun 10, 2016 at 4:19 PM, Boris Pavlovic  wrote:

> Morgan,
>
>
>> When there were failures, the failures were both not looked at by the
>> Rally team and was not performance reasons at the time, it was rally not
>> able to be setup/run at all.
>
>
> Prove this, or it's not true. I agree there were such situations (very
> rarely actually) and we were fixing them quickly (because we have voting
> jobs in our CI pipelines), so in most cases Rally didn't failed because
> it's Rally issue.
>
>
>
I do remember it was not strictly "keystone performance" but there was a
good chunk of time where rally didn't even run because it wasn't able to
standup within keystone, which drove the original (may of 2015) proposal to
drop rally tests. Unfortunately we do not store logs back that far.


> OSProfiler etc had security concerns and issues that were basically left
>> in "review state" after being given clear "do X to have it approved". I
>> want to point out that once the performance team came back and addressed
>> the issues we landed support for OSProfiler, and it is in keystone. It is
>> not enabled by default (profiling should be opt in, and I stand by that),
>> but you are correct we landed it.
>
>
> There were no secure issues, since we introduced HMAC approach which was
> implemented in June 2014 *// 2 years ago. *
> -
> https://github.com/openstack/osprofiler/commit/76a99f2ccc32ea4426717333bbb75fb8b533
>
>

>
- As well you were able to disable osprofiler since
> https://github.com/openstack/osprofiler/commit/988909f112ffe79f8855c4713c2c791dd274bb8d
>  *Jul
> 2014 // 2 years ago*
>
>
So for some reason it takes so long to add at least something to keystone
> code based (by the way osprofiler team was forced to modify it very very
> bad way)
>

The request was simply to make it disabled by default, which was what
you're claiming is the "bad" modification?. I'll argue that OSProfiler
should *never* in it's code default to on. It should always 100% of the
time be opt-in. The gate and everywhere that you want profiling it is fine
to opt in. A production system should never have profiling code enabled
without the operator/deployer explicitly turning it on / adding it in. The
default-on was a very large driver for holding back on the changes. I want
to say that OSProfiler supported default-off was where I was willing to
accept it into our tree (and it did land).


>

> I'm super happy to see a consistent report leaving data about performance,
>> specifically in a consistent environment that isn't going to vary massively
>> between runs (hopefully). Longterm I'd like to also see this [if it isn't
>> already] do a delta-over-time of keystone performance on merged patches, so
>> we can see the timeline of performance.
>
>
> By the way, Rally stores all results to DB (it stored it always) and is
> know capable to build performance trends of many runs
>
>
I would like to still see rally store this centrally and provide the useful
delta data to the developers. This was my request to make it easier to
understand the Rally data. Capable of doing so is different than actually
doing so. If this becomes a reality with rally (and consistent
hardware/node-profile) I will 100% support adding it as a check job back
in. I do think it is likely the rally job to be really useful for
developers would need to be 3rd party; this is because the control of the
types of nodes/performance of said nodes, varies drastically across
providers and AZs.

The work Lance is doing may also lean on rally at some point (I'm not being
prescriptive of the technology that should be used for this performance
feedback loop, just that is is a proper performance feedback loop).  In
fact, I encourage you to work with him if Rally is the right tool for this
job! I see Rally as valuable when it can be run in a way to make it easy
for developers (new and continuing contributors) as well as reviewers to
consume the data. I'm happy to run rally locally, but It doesn't mean we
will be seeing that type of effort from every contributor, it is also not
reasonable to ask cores to download the patch and run rally on each one
before/after - reviewers are a bottleneck as is.

As my final point of what would have shifted things for rally within
keystone, I'm going to use cinder output as an example:

This is very nice data [1]. But it shows *only* checks after the patch has
been merged into the repository (as far as I can tell). This is cool! But
what would make it simply more useful as a reviewer or developer from my
perspective is: Run the rally jobs on the data pre-merge, then run it
post-merge [on a clean db/run/etc] of the patch. Give me a difference. Add
in the longer delta-over-time trending in a central location, and we're
talking something amazingly useful and easy for the reviewers and
developers to consume.

[1]

[openstack-dev] [ironic] IPA example hardware managers

2016-06-10 Thread Jay Faulkner

Hi all,

I've been promising to do more knowledge sharing on IPA Hardware 
Managers, specifically in the form of a presentation. However, I wanted 
to go a different route that would be more likely to stand the test of 
time and be more self-service.


To that end, I've created a couple of well-commented example hardware 
managers here: 
https://github.com/jayofdoom/ipa-example-hardware-managers. If you've 
been wanting to know about how to write additional IPA Hardware 
Managers, between this and the already-existing inline documentation in 
IPA itself there should be more than enough information to get started. 
PRs and feedback accepted.


As a note, my hope is that we can find a reasonable place for this to 
live in the openstack namespace. I only created this in github so I 
could spend my time this afternoon writing the examples rather than 
getting repositories created.


While working on this, however, I came to a realization -- despite our 
documentation telling folks to subclass *either* 
hardware.HardwareManager or hardware.GenericHardwareManager 
(http://docs.openstack.org/developer/ironic-python-agent/#how-can-i-build-a-custom-hardwaremanager), 
after a lot of thought (and some time spent trying to drum up a use case 
for an example where it's useful), I think we should change this, and 
only encourage subclassing HardwareManager for out of tree hardware 
managers. I don't believe there's a technical way to prevent it, so it 
shouldn't technically be an API break, but I wanted to get a consensus 
before moving forward and making that change.


I hope the information is useful!

Thanks,
Jay Faulkner
OSIC


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [qa] Tempest unstable interfaces in plugins

2016-06-10 Thread Assaf Muller
On Fri, Jun 10, 2016 at 12:02 PM, Andrea Frittoli
 wrote:
> Dear all,
>
> I'm working on making the client manager in Tempest a stable interface, so
> that in future it may be used safely by plugins to easily gain access
> service clients [0].
>
> This work inevitably involves changing the current client manager (unstable)
> interface.
> Several tempest plugins in OpenStack already consume that interface (namely
> the manager.Manager class) [1], so my work is likely to break them.
>
> I would ask the people maintaining the plugins to be careful about using
> unstable interfaces, as they are likely to change, especially since we're
> working on converting them to stable.
>
> If you maintain a plugin (in OpenStack or outside of OpenStack) that is
> likely to be affected by my work, please keep an eye on my gerrit review
> [0], leave a comment there or ping me on IRC (andreaf), I would very much
> like to make sure the transition is as painless as possible for everyone.

FWIW this doesn't seem to break Neutron:
https://review.openstack.org/#/c/328398/.

I would appreciate it if changes are made in a backwards compatible
manner (Similar to this:
https://review.openstack.org/#/c/322492/13/tempest/common/waiters.py)
so that projects with Tempest plugins may adapt and not break voting
jobs. The reason projects are using interfaces outside of tempest.lib
is that that's all there is, and the alternative of copy/pasting in to
the repo isn't amazing.

>
> andrea
>
> [0] https://review.openstack.org/#/c/326683/
> [1]
> http://codesearch.openstack.org/?q=from%20tempest%20import%20manager=nope==
>
> IRC: andreaf
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

2016-06-10 Thread Boris Pavlovic
Morgan,


> When there were failures, the failures were both not looked at by the
> Rally team and was not performance reasons at the time, it was rally not
> able to be setup/run at all.


Prove this, or it's not true. I agree there were such situations (very
rarely actually) and we were fixing them quickly (because we have voting
jobs in our CI pipelines), so in most cases Rally didn't failed because
it's Rally issue.


OSProfiler etc had security concerns and issues that were basically left in
> "review state" after being given clear "do X to have it approved". I want
> to point out that once the performance team came back and addressed the
> issues we landed support for OSProfiler, and it is in keystone. It is not
> enabled by default (profiling should be opt in, and I stand by that), but
> you are correct we landed it.


There were no secure issues, since we introduced HMAC approach which was
implemented in June 2014 *// 2 years ago. *
-
https://github.com/openstack/osprofiler/commit/76a99f2ccc32ea4426717333bbb75fb8b533

- As well you were able to disable osprofiler since
https://github.com/openstack/osprofiler/commit/988909f112ffe79f8855c4713c2c791dd274bb8d
*Jul
2014 // 2 years ago*

So for some reason it takes so long to add at least something to keystone
code based (by the way osprofiler team was forced to modify it very very
bad way)



I'm super happy to see a consistent report leaving data about performance,
> specifically in a consistent environment that isn't going to vary massively
> between runs (hopefully). Longterm I'd like to also see this [if it isn't
> already] do a delta-over-time of keystone performance on merged patches, so
> we can see the timeline of performance.


By the way, Rally stores all results to DB (it stored it always) and is
know capable to build performance trends of many runs

Best regards,
Boris Pavlovic

>
>
>
>
>
On Fri, Jun 10, 2016 at 3:58 PM, Morgan Fainberg 
wrote:

>
>
> On Fri, Jun 10, 2016 at 3:26 PM, Lance Bragstad 
> wrote:
>
>>
>>1. I care about performance. I just believe that a big hurdle has
>>been finding infrastructure that allows us to run performance tests in a
>>consistent manner. Dedicated infrastructure plays a big role in this,
>> which is hard (if not impossible) to obtain in the gate - making the gate
>>a suboptimal place for performance testing. Consistency is also an issue
>>because the gate is comprised of resources donated from several different
>>providers. Matt lays this out pretty well in his reply above. This sounds
>>like a TODO to hook rally into the keystone-performance/ansible pipeline,
>>then we would have rally and keystone running on bare metal.
>>
>> This was one of the BIGGEST reasons rally was not given much credence in
> keystone. The wild variations made the rally data mostly noise. We can't
> even tell if the data from similar nodes (same provider/same az) was
> available. This made it a best guess effort of "is this an issue with a
> node being slow, or the patch" at the time the gate was enabled. This is
> also why I wouldn't support re-enabling rally as an in-infra gate/check
> job. The data was extremely difficult to consume as a developer because I'd
> have to either directly run rally here locally (fine, but why waste infra
> resources then?) or try and correlate data from across different patches
> and different AZ providers. It's great to see this being addressed here.
>
>>
>>1.
>>2. See response to #5.
>>3. What were the changes made to keystone that caused rally to fail?
>>If you have some links I'd be curious to revisit them and improve them if 
>> I
>>can.
>>
>> When there were failures, the failures were both not looked at by the
> Rally team and was not performance reasons at the time, it was rally not
> able to be setup/run at all.
>
>>
>>1. Blocked because changes weren't reviewed? As far as I know
>>OSProfiler is in keystone's default pipeline.
>>
>> OSProfiler etc had security concerns and issues that were basically left
> in "review state" after being given clear "do X to have it approved". I
> want to point out that once the performance team came back and addressed
> the issues we landed support for OSProfiler, and it is in keystone. It is
> not enabled by default (profiling should be opt in, and I stand by that),
> but you are correct we landed it.
>
>>
>>1. It doesn't look like there are any open patches for rally
>>integration with keystone [0]. The closed ones have either been
>>merged [1][2][3][4] or abandon [5][6][7][8] because they are
>>work-in-progress or unattended.
>>
>> I'm only looking for this bot to leave a comment. I don't intend on it
>> being a voting job any time soon, it's just providing a datapoint for
>> patches that we suspect to have an impact on performance. It's running on
>> dedicated hardware, but only from a single service provider - so mileage

Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

2016-06-10 Thread Boris Pavlovic
Lance,

I share just how it looked from my side.
I really support your idea (no matter what you pick to use your
tooling/rally/jmeter) it is very valuable, especially if it will become
voting job.
This really should be done by someone.


Best regards,
Boris Pavlovic

On Fri, Jun 10, 2016 at 3:26 PM, Lance Bragstad  wrote:

>
>1. I care about performance. I just believe that a big hurdle has been
>finding infrastructure that allows us to run performance tests in a
>consistent manner. Dedicated infrastructure plays a big role in this,
> which is hard (if not impossible) to obtain in the gate - making the gate
>a suboptimal place for performance testing. Consistency is also an issue
>because the gate is comprised of resources donated from several different
>providers. Matt lays this out pretty well in his reply above. This sounds
>like a TODO to hook rally into the keystone-performance/ansible pipeline,
>then we would have rally and keystone running on bare metal.
>2. See response to #5.
>3. What were the changes made to keystone that caused rally to fail?
>If you have some links I'd be curious to revisit them and improve them if I
>can.
>4. Blocked because changes weren't reviewed? As far as I know
>OSProfiler is in keystone's default pipeline.
>5. It doesn't look like there are any open patches for rally
>integration with keystone [0]. The closed ones have either been
>merged [1][2][3][4] or abandon [5][6][7][8] because they are
>work-in-progress or unattended.
>
> I'm only looking for this bot to leave a comment. I don't intend on it
> being a voting job any time soon, it's just providing a datapoint for
> patches that we suspect to have an impact on performance. It's running on
> dedicated hardware, but only from a single service provider - so mileage
> may vary depending on where and how you run keystone. But, it does take us
> a step in the right direction. People don't have to use it if they don't
> want to.
>
> Thanks for the feedback!
>
> [0]
> https://review.openstack.org/#/q/project:openstack/keystone+message:%22%255E%2540rally%2540%22
> [1] https://review.openstack.org/#/c/240251/
> [2] https://review.openstack.org/#/c/188457/
> [3] https://review.openstack.org/#/c/188352/
> [4] https://review.openstack.org/#/c/90405/
> [5] https://review.openstack.org/#/c/301367/
> [6] https://review.openstack.org/#/c/188479/
> [7] https://review.openstack.org/#/c/98836/
> [8] https://review.openstack.org/#/c/91677/
>
> On Fri, Jun 10, 2016 at 4:26 PM, Boris Pavlovic  wrote:
>
>> Lance,
>>
>> It is amazing effort, I am wishing you good luck with Keystone team,
>> however i faced some issues when I started similar effort
>> about 3 years ago with Rally. Here are some points, that are going to be
>> very useful for you:
>>
>>1. I think that Keystone team doesn't care about performance &
>>scalability at all
>>2. Keystone team ignored/discard all help from Rally team to make
>>this effort successful
>>3. When Rally job started failing, because of introduced performance
>>issues in Keystone, they decided to remove job
>>4. They blocked almost forever work on OSProfiler so we are blind and
>>can't see where is the issue in code
>>5. They didn't help to develop any Rally plugin or even review the
>>Rally test cases that we proposed to them
>>
>>
>> Best regards,
>> Boris Pavlovic
>>
>> On Mon, Jun 6, 2016 at 10:45 AM, Clint Byrum  wrote:
>>
>>> Excerpts from Brant Knudson's message of 2016-06-03 15:16:20 -0500:
>>> > On Fri, Jun 3, 2016 at 2:35 PM, Lance Bragstad 
>>> wrote:
>>> >
>>> > > Hey all,
>>> > >
>>> > > I have been curious about impact of providing performance feedback
>>> as part
>>> > > of the review process. From what I understand, keystone used to have
>>> a
>>> > > performance job that would run against proposed patches (I've only
>>> heard
>>> > > about it so someone else will have to keep me honest about its
>>> timeframe),
>>> > > but it sounds like it wasn't valued.
>>> > >
>>> > >
>>> > We had a job running rally for a year (I think) that nobody ever
>>> looked at
>>> > so we decided it was a waste and stopped running it.
>>> >
>>> > > I think revisiting this topic is valuable, but it raises a series of
>>> > > questions.
>>> > >
>>> > > Initially it probably only makes sense to test a reasonable set of
>>> > > defaults. What do we want these defaults to be? Should they be
>>> determined
>>> > > by DevStack, openstack-ansible, or something else?
>>> > >
>>> > >
>>> > A performance test is going to depend on the environment (the machines,
>>> > disks, network, etc), the existing data (tokens, revocations, users,
>>> etc.),
>>> > and the config (fernet, uuid, caching, etc.). If these aren't
>>> consistent
>>> > between runs then the results are not going to be usable. (This is the
>>> > problem with running 

Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

2016-06-10 Thread Morgan Fainberg
On Fri, Jun 10, 2016 at 3:26 PM, Lance Bragstad  wrote:

>
>1. I care about performance. I just believe that a big hurdle has been
>finding infrastructure that allows us to run performance tests in a
>consistent manner. Dedicated infrastructure plays a big role in this,
> which is hard (if not impossible) to obtain in the gate - making the gate
>a suboptimal place for performance testing. Consistency is also an issue
>because the gate is comprised of resources donated from several different
>providers. Matt lays this out pretty well in his reply above. This sounds
>like a TODO to hook rally into the keystone-performance/ansible pipeline,
>then we would have rally and keystone running on bare metal.
>
> This was one of the BIGGEST reasons rally was not given much credence in
keystone. The wild variations made the rally data mostly noise. We can't
even tell if the data from similar nodes (same provider/same az) was
available. This made it a best guess effort of "is this an issue with a
node being slow, or the patch" at the time the gate was enabled. This is
also why I wouldn't support re-enabling rally as an in-infra gate/check
job. The data was extremely difficult to consume as a developer because I'd
have to either directly run rally here locally (fine, but why waste infra
resources then?) or try and correlate data from across different patches
and different AZ providers. It's great to see this being addressed here.

>
>1.
>2. See response to #5.
>3. What were the changes made to keystone that caused rally to fail?
>If you have some links I'd be curious to revisit them and improve them if I
>can.
>
> When there were failures, the failures were both not looked at by the
Rally team and was not performance reasons at the time, it was rally not
able to be setup/run at all.

>
>1. Blocked because changes weren't reviewed? As far as I know
>OSProfiler is in keystone's default pipeline.
>
> OSProfiler etc had security concerns and issues that were basically left
in "review state" after being given clear "do X to have it approved". I
want to point out that once the performance team came back and addressed
the issues we landed support for OSProfiler, and it is in keystone. It is
not enabled by default (profiling should be opt in, and I stand by that),
but you are correct we landed it.

>
>1. It doesn't look like there are any open patches for rally
>integration with keystone [0]. The closed ones have either been
>merged [1][2][3][4] or abandon [5][6][7][8] because they are
>work-in-progress or unattended.
>
> I'm only looking for this bot to leave a comment. I don't intend on it
> being a voting job any time soon, it's just providing a datapoint for
> patches that we suspect to have an impact on performance. It's running on
> dedicated hardware, but only from a single service provider - so mileage
> may vary depending on where and how you run keystone. But, it does take us
> a step in the right direction. People don't have to use it if they don't
> want to.
>
>
I'm super happy to see a consistent report leaving data about performance,
specifically in a consistent environment that isn't going to vary massively
between runs (hopefully). Longterm I'd like to also see this [if it isn't
already] do a delta-over-time of keystone performance on merged patches, so
we can see the timeline of performance.


> Thanks for the feedback!
>
> [0]
> https://review.openstack.org/#/q/project:openstack/keystone+message:%22%255E%2540rally%2540%22
> [1] https://review.openstack.org/#/c/240251/
> [2] https://review.openstack.org/#/c/188457/
> [3] https://review.openstack.org/#/c/188352/
> [4] https://review.openstack.org/#/c/90405/
> [5] https://review.openstack.org/#/c/301367/
> [6] https://review.openstack.org/#/c/188479/
> [7] https://review.openstack.org/#/c/98836/
> [8] https://review.openstack.org/#/c/91677/
>
>
Great work lance!

--Morgan


> On Fri, Jun 10, 2016 at 4:26 PM, Boris Pavlovic  wrote:
>
>> Lance,
>>
>> It is amazing effort, I am wishing you good luck with Keystone team,
>> however i faced some issues when I started similar effort
>> about 3 years ago with Rally. Here are some points, that are going to be
>> very useful for you:
>>
>>1. I think that Keystone team doesn't care about performance &
>>scalability at all
>>2. Keystone team ignored/discard all help from Rally team to make
>>this effort successful
>>3. When Rally job started failing, because of introduced performance
>>issues in Keystone, they decided to remove job
>>4. They blocked almost forever work on OSProfiler so we are blind and
>>can't see where is the issue in code
>>5. They didn't help to develop any Rally plugin or even review the
>>Rally test cases that we proposed to them
>>
>>
>> Best regards,
>> Boris Pavlovic
>>
>> On Mon, Jun 6, 2016 at 10:45 AM, Clint Byrum  wrote:

Re: [openstack-dev] [Higgins] Call for contribution for Higgins API design

2016-06-10 Thread Hongbin Lu
Yuanying,

The etherpads you pointed to were a few years ago and the information looks a 
bit outdated. I think we can collaborate a similar etherpad with updated 
information (i.e. remove container runtimes that we don’t care, add container 
runtimes that we care). The existing etherpad can be used as a starting point. 
What do you think?

Best regards,
Hongbin

From: Yuanying OTSUKA [mailto:yuany...@oeilvert.org]
Sent: June-01-16 12:43 AM
To: OpenStack Development Mailing List (not for usage questions); Sheel Rana 
Insaan
Cc: adit...@nectechnologies.in; yanya...@cn.ibm.com; flw...@catalyst.net.nz; Qi 
Ming Teng; sitlani.namr...@yahoo.in; Yuanying; Chandan Kumar
Subject: Re: [openstack-dev] [Higgins] Call for contribution for Higgins API 
design

Just F.Y.I.

When Magnum wanted to become “Container as a Service”,
There were some discussion about API design.

* https://etherpad.openstack.org/p/containers-service-api
* https://etherpad.openstack.org/p/openstack-containers-service-api



2016年6月1日(水) 12:09 Hongbin Lu 
>:
Sheel,

Thanks for taking the responsibility. Assigned the BP to you. As discussed, 
please submit a spec for the API design. Feel free to let us know if you need 
any help.

Best regards,
Hongbin

From: Sheel Rana Insaan 
[mailto:ranasheel2...@gmail.com]
Sent: May-31-16 9:23 PM
To: Hongbin Lu
Cc: adit...@nectechnologies.in; 
vivek.jain.openst...@gmail.com; 
flw...@catalyst.net.nz; Shuu Mutou; Davanum 
Srinivas; OpenStack Development Mailing List (not for usage questions); Chandan 
Kumar; hai...@xr.jp.nec.com; Qi Ming Teng; 
sitlani.namr...@yahoo.in; Yuanying; Kumari, 
Madhuri; yanya...@cn.ibm.com
Subject: Re: [Higgins] Call for contribution for Higgins API design


Dear Hongbin,

I am interested in this.
Thanks!!

Best Regards,
Sheel Rana
On Jun 1, 2016 3:53 AM, "Hongbin Lu" 
> wrote:
Hi team,

As discussed in the last team meeting, we agreed to define core use cases for 
the API design. I have created a blueprint for that. We need an owner of the 
blueprint and it requires a spec to clarify the API design. Please let me know 
if you interest in this work (it might require a significant amount of time to 
work on the spec).

https://blueprints.launchpad.net/python-higgins/+spec/api-design

Best regards,
Hongbin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [javascript] Meeting time doodle

2016-06-10 Thread Michael Krotscheck
Alright, the first meeting will be on monday, 1500UTC. For the first
meeting we'll just meet in #openstack-javascript, and see which of the
rooms are available at that time. The agenda is here, go ahead and add
anything you'd think is pertinent:

https://etherpad.openstack.org/p/javascript-meeting-agenda

Michael

On Mon, Jun 6, 2016 at 11:34 AM Michael Krotscheck 
wrote:

> Between fuel, ironic, horizon, storyboard, the app ecosystem group, the
> partridges, the pear trees, and the kitchen sinks, there's an awful lot of
> JavaScript work happening in OpenStack. Enough so that it's a good idea to
> actually start having regular about it.
>
> I've tried to identify dates/times in the week when a meeting channel
> might be open. If you work, consume, and/or want to contribute to
> JavaScript in OpenStack, please fill out this doodle and let me know when
> you can attend!
>
> http://doodle.com/poll/3hxubef6va5wzpkc
>
> Michael Krotscheck
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

2016-06-10 Thread Lance Bragstad
   1. I care about performance. I just believe that a big hurdle has been
   finding infrastructure that allows us to run performance tests in a
   consistent manner. Dedicated infrastructure plays a big role in this,
which is hard (if not impossible) to obtain in the gate - making the gate
   a suboptimal place for performance testing. Consistency is also an issue
   because the gate is comprised of resources donated from several different
   providers. Matt lays this out pretty well in his reply above. This sounds
   like a TODO to hook rally into the keystone-performance/ansible pipeline,
   then we would have rally and keystone running on bare metal.
   2. See response to #5.
   3. What were the changes made to keystone that caused rally to fail? If
   you have some links I'd be curious to revisit them and improve them if I
   can.
   4. Blocked because changes weren't reviewed? As far as I know OSProfiler
   is in keystone's default pipeline.
   5. It doesn't look like there are any open patches for rally integration
   with keystone [0]. The closed ones have either been merged [1][2][3][4] or
   abandon [5][6][7][8] because they are work-in-progress or unattended.

I'm only looking for this bot to leave a comment. I don't intend on it
being a voting job any time soon, it's just providing a datapoint for
patches that we suspect to have an impact on performance. It's running on
dedicated hardware, but only from a single service provider - so mileage
may vary depending on where and how you run keystone. But, it does take us
a step in the right direction. People don't have to use it if they don't
want to.

Thanks for the feedback!

[0]
https://review.openstack.org/#/q/project:openstack/keystone+message:%22%255E%2540rally%2540%22
[1] https://review.openstack.org/#/c/240251/
[2] https://review.openstack.org/#/c/188457/
[3] https://review.openstack.org/#/c/188352/
[4] https://review.openstack.org/#/c/90405/
[5] https://review.openstack.org/#/c/301367/
[6] https://review.openstack.org/#/c/188479/
[7] https://review.openstack.org/#/c/98836/
[8] https://review.openstack.org/#/c/91677/

On Fri, Jun 10, 2016 at 4:26 PM, Boris Pavlovic  wrote:

> Lance,
>
> It is amazing effort, I am wishing you good luck with Keystone team,
> however i faced some issues when I started similar effort
> about 3 years ago with Rally. Here are some points, that are going to be
> very useful for you:
>
>1. I think that Keystone team doesn't care about performance &
>scalability at all
>2. Keystone team ignored/discard all help from Rally team to make this
>effort successful
>3. When Rally job started failing, because of introduced performance
>issues in Keystone, they decided to remove job
>4. They blocked almost forever work on OSProfiler so we are blind and
>can't see where is the issue in code
>5. They didn't help to develop any Rally plugin or even review the
>Rally test cases that we proposed to them
>
>
> Best regards,
> Boris Pavlovic
>
> On Mon, Jun 6, 2016 at 10:45 AM, Clint Byrum  wrote:
>
>> Excerpts from Brant Knudson's message of 2016-06-03 15:16:20 -0500:
>> > On Fri, Jun 3, 2016 at 2:35 PM, Lance Bragstad 
>> wrote:
>> >
>> > > Hey all,
>> > >
>> > > I have been curious about impact of providing performance feedback as
>> part
>> > > of the review process. From what I understand, keystone used to have a
>> > > performance job that would run against proposed patches (I've only
>> heard
>> > > about it so someone else will have to keep me honest about its
>> timeframe),
>> > > but it sounds like it wasn't valued.
>> > >
>> > >
>> > We had a job running rally for a year (I think) that nobody ever looked
>> at
>> > so we decided it was a waste and stopped running it.
>> >
>> > > I think revisiting this topic is valuable, but it raises a series of
>> > > questions.
>> > >
>> > > Initially it probably only makes sense to test a reasonable set of
>> > > defaults. What do we want these defaults to be? Should they be
>> determined
>> > > by DevStack, openstack-ansible, or something else?
>> > >
>> > >
>> > A performance test is going to depend on the environment (the machines,
>> > disks, network, etc), the existing data (tokens, revocations, users,
>> etc.),
>> > and the config (fernet, uuid, caching, etc.). If these aren't consistent
>> > between runs then the results are not going to be usable. (This is the
>> > problem with running rally on infra hardware.) If the data isn't
>> realistic
>> > (1000s of tokens, etc.) then the results are going to be at best not
>> useful
>> > or at worst misleading.
>> >
>>
>> That's why I started the counter-inspection spec:
>>
>>
>> http://specs.openstack.org/openstack/qa-specs/specs/devstack/counter-inspection.html
>>
>> It just tries to count operations, and graph those. I've, unfortunately,
>> been pulled off to other things of late, but I do intend to loop back
>> and hit this hard 

Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port isActive

2016-06-10 Thread Kevin Benton
Polling should be fine. get_port operations a relatively cheap operation
for Neutron.

Maybe for the future we can have a more pluggable version of the nova
callback notifier we have in Neutron like Salvatore pointed out.

On Fri, Jun 10, 2016 at 7:49 AM, Mohammad Banikazemi  wrote:

> Hi Neil,
>
> Currently, when a docker libnetwork "join" operation in Kuryr is returned,
> it is not guaranteed that the network connectivity has been established.
> There are containers that check for network connectivity as the first thing
> they do when they come up and under heavy load some notice there is no
> connectivity and simply bail out. I am trying to deal with such a use case,
>
> Thanks for pointing out that option 2 won't work for you. I think
> Salvatore also alluded to that in his response. What you are suggesting
> with pinging the container from the appropriate namespace may be worth a
> try but then there may be containers that do not allow ingress traffic
> while they are up and happy. So short of what Salvatore suggested in his
> earlier email (and I am not sure if that can be done without additions to
> Neutron), we are left with option 1.
>
> Keep in mind that users can choose not to enable the blocking option and
> things will be as they are right now. Would that be reasonable?
>
> Best,
>
> Mohammad
>
> [image: Inactive hide details for Neil Jerram ---06/10/2016 09:25:59
> AM---Hi Mohammad, Why is the blocking needed? Is it to report som]Neil
> Jerram ---06/10/2016 09:25:59 AM---Hi Mohammad, Why is the blocking needed?
> Is it to report some kind of status back to
>
> From: Neil Jerram 
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: 06/10/2016 09:25 AM
> Subject: Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port
> is Active
> --
>
>
>
> Hi Mohammad,
>
> Why is the blocking needed?  Is it to report some kind of status back to
> Docker/Kubernetes, or to allow some follow-on action to happen?
>
> When using networking-calico as the driver, I think that only option (1)
> would work, out of the options you've suggested below.  (3) doesn't work,
> as you say, because Calico doesn't involve an L2 agent.  Also Calico
> doesn't use the RPC message queue for reporting port status, because we've
> found that that message queue is in itself a scalability bottleneck.
>
> I guess another option would be for the using system to determine for
> itself when the port appears to be working, e.g. by the host pinging the
> container/pod's IP address.
>
> Regards,
> Neil
>
>
> On Wed, Jun 8, 2016 at 4:23 PM Mohammad Banikazemi <*m...@us.ibm.com*
> > wrote:
>
>For the Kuryr project, in order to support blocking until vifs are
>plugged in (that is adding config options similar to the following options
>define in Nova: vif_plugging_is_fatal and vif_plugging_timeout), we need to
>detect that the Neutron plugin being used is done with plugging a given 
> vif.
>
>Here are a few options:
>
>1- The simplest approach seems to be polling for the status of the
>Neutron port to become Active. (This may lead to scalability issues but
>short of having a specific goal for scalability, it is not clear that will
>be the case.)
>2- Alternatively, We could subscribe to the message queue and wait for
>such a port update event.
>3- It was also suggested that we could use l2 agent extension to
>detect such an event but that seems to limit us to certain Neutron plugins
>and therefore not acceptable.
>
>I was wondering if there are other and better options.
>
>Best,
>
>Mohammad
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe:
>*openstack-dev-requ...@lists.openstack.org?subject:unsubscribe*
>
> *http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev*
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe:
>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [Neutron][os-vif] Expanding vif capability for wiring trunk ports

2016-06-10 Thread Carl Baldwin
Here's a link directly to the current design proposal [1] that might
be of interest.

[1] 
https://review.openstack.org/#/c/318317/4/doc/source/devref/openvswitch_agent.rst@463

On Thu, Jun 9, 2016 at 5:31 PM, Carl Baldwin  wrote:
> Hi,
>
> You may or may not be aware of the vlan-aware-vms effort [1] in
> Neutron.  If not, there is a spec and a fair number of patches in
> progress for this.  Essentially, the goal is to allow a VM to connect
> to multiple Neutron networks by tagging traffic on a single port with
> VLAN tags.
>
> This effort will have some effect on vif plugging because the datapath
> will include some changes that will effect how vif plugging is done
> today.
>
> The design proposal for trunk ports with OVS adds a new bridge for
> each trunk port.  This bridge will demux the traffic and then connect
> to br-int with patch ports for each of the networks.  Rawlin Peters
> has some ideas for expanding the vif capability to include this
> wiring.
>
> There is also a proposal for connecting to linux bridges by using
> kernel vlan interfaces.
>
> This effort is pretty important to Neutron in the Newton timeframe.  I
> wanted to send this out to start rounding up the reviewers and other
> participants we need to see how we can start putting together a plan
> for nova integration of this feature (via os-vif?).
>
> Carl Baldwin
>
> [1] https://review.openstack.org/#/q/topic:bp/vlan-aware-vms+-status:abandoned

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Jim Rollenhagen
On Fri, Jun 10, 2016 at 12:35:43PM -0700, Devananda van der Veen wrote:
> On 06/10/2016 05:48 AM, Sean Dague wrote:
> > On 06/10/2016 08:41 AM, Jeremy Stanley wrote:
> >> On 2016-06-10 11:49:12 +0100 (+0100), Miles Gould wrote:
> >>> On 09/06/16 23:21, Jay Faulkner wrote:
>  There was some discussion about whether or not the Ironic grenade job
>  should be in the check pipeline (even as -nv) for grenade,
> >>>
> >>> Not having this would mean that changes to grenade could silently break
> >>> Ironic's CI, right? That sounds really bad.
> >>
> >> That's like saying it's really bad that changes to devstack could
> >> silently break devstack-based jobs for random projects, and so they
> >> should be tested against every one of those jobs. At some point you
> >> have to draw a line between running a reasonably representative
> >> sample and running so many jobs that you'll never be able to merge
> >> another change again (because even very small nondeterministic
> >> failure rates compound to make that impossible at a certain scale).
> > 
> > Nothing should be voting in check in grenade that requires a plugin.
> > 
> > I'm fine with a few things in check nv if they are doing something out
> > of the ordinary that we think needs to be kept on top of. I also expect
> > that ironic folks are going to watch for those failures, and say, with
> > -1/+1 CR, when they are legit and when it was off the rails. A non
> > voting job that doesn't have domain experts validating the content
> > regularly with CR means it gets ignored if it fails a bunch.
> > 
> 
> I'd like to see this job running in the grenade check queue so we can watch it
> there, and trace back to anything that affects us in unexpected ways. As Sean
> points out, it should not be made voting in grenade's check queue.

FWIW, Sean approved that this morning :)

> 
> It _should_ be made voting in Ironic's queue as soon as we have gathered
> stability data for it. I'd love to see that get turned on in a week. With the
> current patch volume, I think we'll be able to get plenty of stability data in
> that time.

++ agree completely.

// jim

> 
> --devananda
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port isActive

2016-06-10 Thread Antoni Segura Puimedon
On Fri, Jun 10, 2016 at 5:18 PM, Neil Jerram  wrote:

> Yes, that all makes sense - thanks for explaining.
>
> Neil
>
>
> On Fri, Jun 10, 2016 at 3:50 PM Mohammad Banikazemi  wrote:
>
>> Hi Neil,
>>
>> Currently, when a docker libnetwork "join" operation in Kuryr is
>> returned, it is not guaranteed that the network connectivity has been
>> established. There are containers that check for network connectivity as
>> the first thing they do when they come up and under heavy load some notice
>> there is no connectivity and simply bail out. I am trying to deal with such
>> a use case,
>>
>> Thanks for pointing out that option 2 won't work for you. I think
>> Salvatore also alluded to that in his response. What you are suggesting
>> with pinging the container from the appropriate namespace may be worth a
>> try but then there may be containers that do not allow ingress traffic
>> while they are up and happy. So short of what Salvatore suggested in his
>> earlier email (and I am not sure if that can be done without additions to
>> Neutron), we are left with option 1.
>>
>> Keep in mind that users can choose not to enable the blocking option and
>> things will be as they are right now. Would that be reasonable?
>>
>
That means going for the current version in the patch, right?


>
>> Best,
>>
>> Mohammad
>>
>> [image: Inactive hide details for Neil Jerram ---06/10/2016 09:25:59
>> AM---Hi Mohammad, Why is the blocking needed? Is it to report som]Neil
>> Jerram ---06/10/2016 09:25:59 AM---Hi Mohammad, Why is the blocking needed?
>> Is it to report some kind of status back to
>>
>> From: Neil Jerram 
>> To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev@lists.openstack.org>
>> Date: 06/10/2016 09:25 AM
>>
>>
>> Subject: Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron
>> Port is Active
>> --
>>
>>
>>
>> Hi Mohammad,
>>
>> Why is the blocking needed?  Is it to report some kind of status back to
>> Docker/Kubernetes, or to allow some follow-on action to happen?
>>
>> When using networking-calico as the driver, I think that only option (1)
>> would work, out of the options you've suggested below.  (3) doesn't work,
>> as you say, because Calico doesn't involve an L2 agent.  Also Calico
>> doesn't use the RPC message queue for reporting port status, because we've
>> found that that message queue is in itself a scalability bottleneck.
>>
>> I guess another option would be for the using system to determine for
>> itself when the port appears to be working, e.g. by the host pinging the
>> container/pod's IP address.
>>
>> Regards,
>> Neil
>>
>>
>> On Wed, Jun 8, 2016 at 4:23 PM Mohammad Banikazemi <*m...@us.ibm.com*
>> > wrote:
>>
>>
>>For the Kuryr project, in order to support blocking until vifs are
>>plugged in (that is adding config options similar to the following options
>>define in Nova: vif_plugging_is_fatal and vif_plugging_timeout), we need 
>> to
>>detect that the Neutron plugin being used is done with plugging a given 
>> vif.
>>
>>Here are a few options:
>>
>>1- The simplest approach seems to be polling for the status of the
>>Neutron port to become Active. (This may lead to scalability issues but
>>short of having a specific goal for scalability, it is not clear that will
>>be the case.)
>>2- Alternatively, We could subscribe to the message queue and wait
>>for such a port update event.
>>3- It was also suggested that we could use l2 agent extension to
>>detect such an event but that seems to limit us to certain Neutron plugins
>>and therefore not acceptable.
>>
>>I was wondering if there are other and better options.
>>
>>Best,
>>
>>Mohammad
>>
>>__
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe:
>>*openstack-dev-requ...@lists.openstack.org?subject:unsubscribe*
>>
>>
>>
>>*http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev*
>>
>>__
>>
>>
>>
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe:
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>
>>
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __

Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

2016-06-10 Thread Boris Pavlovic
Lance,

It is amazing effort, I am wishing you good luck with Keystone team,
however i faced some issues when I started similar effort
about 3 years ago with Rally. Here are some points, that are going to be
very useful for you:

   1. I think that Keystone team doesn't care about performance &
   scalability at all
   2. Keystone team ignored/discard all help from Rally team to make this
   effort successful
   3. When Rally job started failing, because of introduced performance
   issues in Keystone, they decided to remove job
   4. They blocked almost forever work on OSProfiler so we are blind and
   can't see where is the issue in code
   5. They didn't help to develop any Rally plugin or even review the Rally
   test cases that we proposed to them


Best regards,
Boris Pavlovic

On Mon, Jun 6, 2016 at 10:45 AM, Clint Byrum  wrote:

> Excerpts from Brant Knudson's message of 2016-06-03 15:16:20 -0500:
> > On Fri, Jun 3, 2016 at 2:35 PM, Lance Bragstad 
> wrote:
> >
> > > Hey all,
> > >
> > > I have been curious about impact of providing performance feedback as
> part
> > > of the review process. From what I understand, keystone used to have a
> > > performance job that would run against proposed patches (I've only
> heard
> > > about it so someone else will have to keep me honest about its
> timeframe),
> > > but it sounds like it wasn't valued.
> > >
> > >
> > We had a job running rally for a year (I think) that nobody ever looked
> at
> > so we decided it was a waste and stopped running it.
> >
> > > I think revisiting this topic is valuable, but it raises a series of
> > > questions.
> > >
> > > Initially it probably only makes sense to test a reasonable set of
> > > defaults. What do we want these defaults to be? Should they be
> determined
> > > by DevStack, openstack-ansible, or something else?
> > >
> > >
> > A performance test is going to depend on the environment (the machines,
> > disks, network, etc), the existing data (tokens, revocations, users,
> etc.),
> > and the config (fernet, uuid, caching, etc.). If these aren't consistent
> > between runs then the results are not going to be usable. (This is the
> > problem with running rally on infra hardware.) If the data isn't
> realistic
> > (1000s of tokens, etc.) then the results are going to be at best not
> useful
> > or at worst misleading.
> >
>
> That's why I started the counter-inspection spec:
>
>
> http://specs.openstack.org/openstack/qa-specs/specs/devstack/counter-inspection.html
>
> It just tries to count operations, and graph those. I've, unfortunately,
> been pulled off to other things of late, but I do intend to loop back
> and hit this hard over the next few months to try and get those graphs.
>
> What we'd get initially is just graphs of how many messages we push
> through RabbitMQ, and how many rows/queries/transactions we push through
> mysql. We may also want to add counters like how many API requests
> happened, and how many retries happen inside the code itself.
>
> There's a _TON_ we can do now to ensure that we know what the trends are
> when something gets "slow", so we can look for a gradual "death by 1000
> papercuts" trend or a hockey stick that can be tied to a particular
> commit.
>
> > What does the performance test criteria look like and where does it live?
> > > Does it just consist of running tempest?
> > >
> > >
> > I don't think tempest is going to give us numbers that we're looking for
> > for performance. I've seen a few scripts and have my own for testing
> > performance of token validation, token creation, user creation, etc.
> which
> > I think will do the exact tests we want and we can get the results
> > formatted however we like.
> >
>
> Agreed that tempest will only give a limited view. Ideally one would
> also test things like "after we've booted 1000 vms, do we end up reading
> 1000 more rows, or 1000 * 1000 more rows.
>
> > From a contributor and reviewer perspective, it would be nice to have the
> > > ability to compare performance results across patch sets. I understand
> that
> > > keeping all performance results for every patch for an extended period
> of
> > > time is unrealistic. Maybe we take a daily performance snapshot against
> > > master and use that to map performance patterns over time?
> > >
> > >
> > Where are you planning to store the results?
> >
>
> Infra has a graphite/statsd cluster which is made for collecting metrics
> on tests. It might need to be expanded a bit, but it should be
> relatively cheap to do so given the benefit of having some of these
> numbers.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack 

Re: [openstack-dev] [keystone][all] Incorporating performance feedback into the review process

2016-06-10 Thread Lance Bragstad
Ok - here is what I have so far [0], and I admit there is still a bunch of
work to do [1]. I encourage folks to poke through the code and suggest
improvements via Github Issues. I've never really stood up third-party
testing before so this is completely new to me and I'm open to feedback,
and working together. I'm trying to document everything I want to do in
Github Issues [2]. Currently - this is only opt-in testing and it will not
vote on a patch. Here is an example of what it looks like in use [3].

This is very much still in the PoC stage. I'm happy to review pull requests
and manage deployments to the testing infrastructure for others who want to
make this better or leverage it to do their own performance testing locally.

Thanks,

Lance
irc: lbragstad

[0] https://github.com/lbragstad/keystone-performance
[1] https://github.com/lbragstad/keystone-performance/issues
[2]
https://github.com/lbragstad/keystone-performance/issues?utf8=%E2%9C%93=is%3Aissue
[3] https://review.openstack.org/#/c/326246/

On Mon, Jun 6, 2016 at 12:45 PM, Clint Byrum  wrote:

> Excerpts from Brant Knudson's message of 2016-06-03 15:16:20 -0500:
> > On Fri, Jun 3, 2016 at 2:35 PM, Lance Bragstad 
> wrote:
> >
> > > Hey all,
> > >
> > > I have been curious about impact of providing performance feedback as
> part
> > > of the review process. From what I understand, keystone used to have a
> > > performance job that would run against proposed patches (I've only
> heard
> > > about it so someone else will have to keep me honest about its
> timeframe),
> > > but it sounds like it wasn't valued.
> > >
> > >
> > We had a job running rally for a year (I think) that nobody ever looked
> at
> > so we decided it was a waste and stopped running it.
> >
> > > I think revisiting this topic is valuable, but it raises a series of
> > > questions.
> > >
> > > Initially it probably only makes sense to test a reasonable set of
> > > defaults. What do we want these defaults to be? Should they be
> determined
> > > by DevStack, openstack-ansible, or something else?
> > >
> > >
> > A performance test is going to depend on the environment (the machines,
> > disks, network, etc), the existing data (tokens, revocations, users,
> etc.),
> > and the config (fernet, uuid, caching, etc.). If these aren't consistent
> > between runs then the results are not going to be usable. (This is the
> > problem with running rally on infra hardware.) If the data isn't
> realistic
> > (1000s of tokens, etc.) then the results are going to be at best not
> useful
> > or at worst misleading.
> >
>
> That's why I started the counter-inspection spec:
>
>
> http://specs.openstack.org/openstack/qa-specs/specs/devstack/counter-inspection.html
>
> It just tries to count operations, and graph those. I've, unfortunately,
> been pulled off to other things of late, but I do intend to loop back
> and hit this hard over the next few months to try and get those graphs.
>
> What we'd get initially is just graphs of how many messages we push
> through RabbitMQ, and how many rows/queries/transactions we push through
> mysql. We may also want to add counters like how many API requests
> happened, and how many retries happen inside the code itself.
>
> There's a _TON_ we can do now to ensure that we know what the trends are
> when something gets "slow", so we can look for a gradual "death by 1000
> papercuts" trend or a hockey stick that can be tied to a particular
> commit.
>
> > What does the performance test criteria look like and where does it live?
> > > Does it just consist of running tempest?
> > >
> > >
> > I don't think tempest is going to give us numbers that we're looking for
> > for performance. I've seen a few scripts and have my own for testing
> > performance of token validation, token creation, user creation, etc.
> which
> > I think will do the exact tests we want and we can get the results
> > formatted however we like.
> >
>
> Agreed that tempest will only give a limited view. Ideally one would
> also test things like "after we've booted 1000 vms, do we end up reading
> 1000 more rows, or 1000 * 1000 more rows.
>
> > From a contributor and reviewer perspective, it would be nice to have the
> > > ability to compare performance results across patch sets. I understand
> that
> > > keeping all performance results for every patch for an extended period
> of
> > > time is unrealistic. Maybe we take a daily performance snapshot against
> > > master and use that to map performance patterns over time?
> > >
> > >
> > Where are you planning to store the results?
> >
>
> Infra has a graphite/statsd cluster which is made for collecting metrics
> on tests. It might need to be expanded a bit, but it should be
> relatively cheap to do so given the benefit of having some of these
> numbers.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: 

Re: [openstack-dev] [Neutron][LBaaS][barbican]TLS container could not be found

2016-06-10 Thread Jiahao Liang
Hi guys,

I was away from Barbican for a couple of month but I still come back has
the same issue.

As mentioned in the original email, I went through
https://wiki.openstack.org/wiki/Network/LBaaS/docs/how-to-create-tls-loadbalancer
with
devstack. And all my branches were set to stable/mitaka.

If I set my user and tenant as "admin admin", the workflow passed. But it
failed if I set the user and tenant to "admin demo", and rerun all the
steps.

Is there anything I was missing from the docs?

Steps to reproduce:

   1. source ~/devstack/openrc admin demo
   2. barbican secret store --payload-content-type='text/plain'
   --name='certificate' --payload="$(cat server.crt)"
   3. barbican secret store --payload-content-type='text/plain'
   --name='private_key' --payload="$(cat server.key)"
   4. barbican secret container create --name='tls_container'
   --type='certificate' --secret="certificate=$(barbican secret list | awk '/
   certificate / {print $2}')" --secret="private_key=$(barbican secret list |
   awk '/ private_key / {print $2}')"
   5. neutron lbaas-loadbalancer-create $(neutron subnet-list | awk '/
   private-subnet / {print $2}') --name lb1
   6. neutron lbaas-listener-create --loadbalancer lb1 --protocol-port 443
   --protocol TERMINATED_HTTPS --name listener1
   --default-tls-container=$(barbican secret container list | awk '/
   tls_container / {print $2}')


The error msg I got is

> $ neutron lbaas-listener-create --loadbalancer
> 738689bd-b54e-485e-b742-57bd6e812270 --protocol-port 443 --protocol
> TERMINATED_HTTPS --name listener2 --default-tls-container=$(barbican secret
> container list | awk '/ tls_container / {print $2}')
> WARNING:barbicanclient.barbican:This Barbican CLI interface has been
> deprecated and will be removed in the O release. Please use the openstack
> unified client instead.
> DEBUG:stevedore.extension:found extension EntryPoint.parse('table =
> cliff.formatters.table:TableFormatter')
> DEBUG:stevedore.extension:found extension EntryPoint.parse('json =
> cliff.formatters.json_format:JSONFormatter')
> DEBUG:stevedore.extension:found extension EntryPoint.parse('csv =
> cliff.formatters.commaseparated:CSVLister')
> DEBUG:stevedore.extension:found extension EntryPoint.parse('value =
> cliff.formatters.value:ValueFormatter')
> DEBUG:stevedore.extension:found extension EntryPoint.parse('yaml =
> cliff.formatters.yaml_format:YAMLFormatter')
> DEBUG:barbicanclient.client:Creating Client object
> DEBUG:barbicanclient.containers:Listing containers - offset 0 limit 10
> name None type None
> DEBUG:keystoneclient.auth.identity.v2:Making authentication request to
> http://192.168.100.148:5000/v2.0/tokens
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
> (1): 192.168.100.148
> Starting new HTTP connection (1): 192.168.100.148
> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
> HTTP/1.1" 200 3924
> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
> http://192.168.100.148:9311 -H "Accept: application/json" -H "User-Agent:
> python-keystoneclient"
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection
> (1): 192.168.100.148
> Starting new HTTP connection (1): 192.168.100.148
> DEBUG:requests.packages.urllib3.connectionpool:"GET / HTTP/1.1" 300 353
> DEBUG:keystoneclient.session:RESP: [300] Content-Length: 353 Content-Type:
> application/json; charset=UTF-8 Connection: close
> RESP BODY: {"versions": {"values": [{"status": "stable", "updated":
> "2015-04-28T00:00:00Z", "media-types": [{"base": "application/json",
> "type": "application/vnd.openstack.key-manager-v1+json"}], "id": "v1",
> "links": [{"href": "http://192.168.100.148:9311/v1/;, "rel": "self"},
> {"href": "http://docs.openstack.org/;, "type": "text/html", "rel":
> "describedby"}]}]}}
> DEBUG:keystoneclient.session:REQ: curl -g -i -X GET
> http://192.168.100.148:9311/v1/containers -H "User-Agent:
> python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token:
> {SHA1}203d7de65f6cfb1fb170437ae2da98fef35f0942"
> INFO:requests.packages.urllib3.connectionpool:Resetting dropped
> connection: 192.168.100.148
> Resetting dropped connection: 192.168.100.148
> DEBUG:requests.packages.urllib3.connectionpool:"GET
> /v1/containers?limit=10=0 HTTP/1.1" 200 585
> DEBUG:keystoneclient.session:RESP: [200] Connection: close Content-Type:
> application/json; charset=UTF-8 Content-Length: 585 x-openstack-request-id:
> req-aa4bb861-3d1d-42c6-be3d-5d3935622043
> RESP BODY: {"total": 1, "containers": [{"status": "ACTIVE", "updated":
> "2016-06-10T01:14:45", "name": "tls_container", "consumers": [], "created":
> "2016-06-10T01:14:45", "container_ref": "
> http://192.168.100.148:9311/v1/containers/4ca420a1-ed23-4e91-a08a-311dad3df801;,
> "creator_id": "9ee7d4959bc74d2988d50e0e3a965c64", "secret_refs":
> [{"secret_ref": "
> http://192.168.100.148:9311/v1/secrets/c96944b3-174e-418f-8598-8979eafaa537;,
> "name": "certificate"}, {"secret_ref": "
> 

Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Devananda van der Veen
On 06/10/2016 05:48 AM, Sean Dague wrote:
> On 06/10/2016 08:41 AM, Jeremy Stanley wrote:
>> On 2016-06-10 11:49:12 +0100 (+0100), Miles Gould wrote:
>>> On 09/06/16 23:21, Jay Faulkner wrote:
 There was some discussion about whether or not the Ironic grenade job
 should be in the check pipeline (even as -nv) for grenade,
>>>
>>> Not having this would mean that changes to grenade could silently break
>>> Ironic's CI, right? That sounds really bad.
>>
>> That's like saying it's really bad that changes to devstack could
>> silently break devstack-based jobs for random projects, and so they
>> should be tested against every one of those jobs. At some point you
>> have to draw a line between running a reasonably representative
>> sample and running so many jobs that you'll never be able to merge
>> another change again (because even very small nondeterministic
>> failure rates compound to make that impossible at a certain scale).
> 
> Nothing should be voting in check in grenade that requires a plugin.
> 
> I'm fine with a few things in check nv if they are doing something out
> of the ordinary that we think needs to be kept on top of. I also expect
> that ironic folks are going to watch for those failures, and say, with
> -1/+1 CR, when they are legit and when it was off the rails. A non
> voting job that doesn't have domain experts validating the content
> regularly with CR means it gets ignored if it fails a bunch.
> 

I'd like to see this job running in the grenade check queue so we can watch it
there, and trace back to anything that affects us in unexpected ways. As Sean
points out, it should not be made voting in grenade's check queue.

It _should_ be made voting in Ironic's queue as soon as we have gathered
stability data for it. I'd love to see that get turned on in a week. With the
current patch volume, I think we'll be able to get plenty of stability data in
that time.

--devananda

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [horizon] 800 UTC Meeting Cancelled on 15/06

2016-06-10 Thread Rob Cresswell
Hi all,

I'm away next week so won't be able to run the 800 UTC meeting. The 2000 UTC 
meeting will be chaired by David Lyle.

Rob
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone] Changing the project name uniqueness constraint

2016-06-10 Thread Morgan Fainberg
On Fri, Jun 10, 2016 at 6:37 AM, Henry Nash  wrote:

> On further reflection, it seems to me that we can never simply enable
> either of these approaches in a single release. Even a v4.0 version of the
> API doesn’t help - since presumably a sever supporting v4 would want to be
> able to support v3.x for a signification time; and, already discussed, as
> soon as you allow multiple none-names to have the same name, you can no
> longer guarantee to support the current API.
>
> Hence the only thing I think we can do (if we really do want to change the
> current functionality) is to do this over several releases with a typical
> deprecation cycle, e.g.
>
> 1) At release 3.7 we allow you to (optionally) specify path names for
> auth….but make no changes to the uniqueness constraints. We also change the
> GET /auth/projects to return a path name. However, you can still auth
> exactly the way we do today (since there will always only be a single
> project of a given node-name). If however, you do auth without a path (to a
> project that isn’t a top level project), we log a warning to say this is
> deprecated (2 cycles, 4 cycles?)
> 2) If you connect with a 3.6 client, then you get the same as today for
> GET /auth/projects and cannot use a path name to auth.
> 3) At sometime in the future, we deprecate the “auth without a path”
> capability. We can debate as to whether this has to be a major release.
>
> If we take this gradual approach, I would be pushing for the “relax
> project name constraints” approach…since I believe this leads to a cleaner
> eventual solution (and there is no particular advantage with the
> hierarchical naming approach) - and (until the end of the deprecation)
> there is no break to the existing API.
>
> Henry
>
>
How do you handle relaxed project name constraints without completely
breaking 3.6 auth - regardless of future microversion that requires the
full path. This is a major api change and will result in very complex
matrix of project name mapping (old projects that can be accessed without
the full path, new that must always have the path)?

Simply put, I do not see relaxing project name constraints as viable
without a major API change and projects that simply are unavailable for
scoping a token to under the base api version (pre-microversions) of 3.6

I am certain that if all projects post API version 3.6 are created with the
full-path name only and that is how they are represented to the user for
auth, we get both things for free. Old projects pre-full-path would need
optional compatibility for deconstructing a full-path for them.  Basically
you end up with "path" and "name", in old projects these differ, in new
projects they are the same.  No conflicts, not breaking "currently working
auth", no "major API version" needed.

--Morgan


> On 7 Jun 2016, at 09:47, Henry Nash  wrote:
>
> OK, so thanks for the feedback - understand the message.
>
> However, in terms of compatibility, the one thing that concerns me about
> the hierarchical naming approach is that even with microversioing, we might
> still surprise a client. An unmodified client (i.e. doesn’t understand 3.7)
> would still see a change in the data being returned (the project names have
> suddenly become full path names). We have to return this even if they don’t
> ask for 3.7, since otherwise there is no difference between this approach
> and relaxing the project naming in terms of trying to prevent auth
> breakages.
>
> In more detail:
>
> 1) Both approaches were planned to return the path name (instead of the
> node name) in GET /auth/projects - i.e. the API you are meant to use to
> find out what you can scope to
> 2) Both approaches were planned to accept the path name in the auth
> request block
> 3) The difference in hierarchical naming is the if I do a regular GET
> /project(s) I also see the full path name as the “project name”
>
> if we don’t do 3), then code that somehow authenticates, and then uses the
> regular GET /project(s) calls to find a project name and then re-scopes (or
> re-auths) to that name, will fail if the project they want is not a top
> level project. However, the flip side is that if there is code that uses
> these same calls to, say, display projects to the user (e.g. a home grown
> UI) - then it might get confused until it supports 3.7 (i.e. asking for the
> old microversion won’t help it) since all the names include the
> hierarchical path.
>
> Just want to make sure we understand the implications….
>
> Henry
>
> On 4 Jun 2016, at 08:34, Monty Taylor  wrote:
>
> On 06/04/2016 01:53 AM, Morgan Fainberg wrote:
>
>
> On Jun 3, 2016 12:42, "Lance Bragstad"  >> wrote:
>
>
>
>
> On Fri, Jun 3, 2016 at 11:20 AM, Henry Nash 
> >> wrote:
>
>
>
> On 3 Jun 2016, at 16:38, Lance Bragstad 

Re: [openstack-dev] [keystone] Changing the project name uniqueness constraint

2016-06-10 Thread Clint Byrum
Excerpts from Henry Nash's message of 2016-06-10 14:37:37 +0100:
> On further reflection, it seems to me that we can never simply enable either 
> of these approaches in a single release. Even a v4.0 version of the API 
> doesn’t help - since presumably a sever supporting v4 would want to be able 
> to support v3.x for a signification time; and, already discussed, as soon as 
> you allow multiple none-names to have the same name, you can no longer 
> guarantee to support the current API.
> 
> Hence the only thing I think we can do (if we really do want to change the 
> current functionality) is to do this over several releases with a typical 
> deprecation cycle, e.g.
> 
> 1) At release 3.7 we allow you to (optionally) specify path names for 
> auth….but make no changes to the uniqueness constraints. We also change the 
> GET /auth/projects to return a path name. However, you can still auth exactly 
> the way we do today (since there will always only be a single project of a 
> given node-name). If however, you do auth without a path (to a project that 
> isn’t a top level project), we log a warning to say this is deprecated (2 
> cycles, 4 cycles?)
> 2) If you connect with a 3.6 client, then you get the same as today for GET 
> /auth/projects and cannot use a path name to auth.
> 3) At sometime in the future, we deprecate the “auth without a path” 
> capability. We can debate as to whether this has to be a major release.
> 
> If we take this gradual approach, I would be pushing for the “relax project 
> name constraints” approach…since I believe this leads to a cleaner eventual 
> solution (and there is no particular advantage with the hierarchical naming 
> approach) - and (until the end of the deprecation) there is no break to the 
> existing API.
> 

This seems really complicated.

Why don't users just start using paths in project names, if they want
paths in project names?

And then in v3.7 you can allow them to specify paths relative to parent of
the user:

So just allow this always:

{"name": "finance/dev"}

And then add this later once users are aware of what the / means:

{"basename": "dev"}

What breaks by adding that?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [swift] crypto-review review, soft-freeze on master

2016-06-10 Thread John Dickinson
As mentioned in IRC and on 
https://wiki.openstack.org/wiki/Swift/PriorityReviews, Swift is now under a 
soft freeze for master.

The patches for the crypto functionality have been proposed to 
feature/crypto-review. The initial plan is to spend two weeks in this soft 
freeze (until June 24) to get the final crypto patches reviewed and landed. 
We'll reevaluate at that time.

* The patch chain allows you to see the steps needed to get to the full crypto 
functionality.
* Please do not push patches over the patch chain, but please leave (links to) 
diffs as review comments.
* acoles is managing this patch chain.
* If there is something that must land on master before feature/crypto-review 
has landed, please let both notmyname and acoles know.
* Once the patches in the chain have been approved , acoles will remove the -2 
on the first patch and allow the chain to land on the crypto-review branch. 
Then we will land a single merge commit to master, bringing in the crypto 
functionality as one commit.

I'm excited to bring this functionality into Swift.


--John




signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [charms] Re-licensing OpenStack charms under Apache 2.0

2016-06-10 Thread Bilal Baqar
Hey James

No problem on my side for this change.

Regards

On Fri, Jun 10, 2016 at 3:25 PM, Neil Jerram  wrote:

> Me too.
>
> On Fri, Jun 10, 2016 at 10:32 AM Cory Benfield  wrote:
>
>>
>> > On 8 Jun 2016, at 11:20, James Page  wrote:
>> >
>> > The majority of contributors are from Canonical (from whom I have
>> permission to make this switch) with a further 18 contributors from outside
>> of Canonical who I will be directly contacting for approval in gerrit as
>> reviews are raised for each repository.
>>
>> Hey James,
>>
>> I’m happy for you to relicense my contributions as Apache 2.0.
>>
>> Cory
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Bilal Baqar
MTS - PLUMgrid Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Random IP address allocations

2016-06-10 Thread Carl Baldwin
On Fri, Jun 10, 2016 at 2:40 AM, Gary Kotton  wrote:
> Hi,
> The patch https://review.openstack.org/#/c/292207/ has broken decomposed
> plugins. I am not sure if we can classify this as a API change – basically

Can you be more specific about how it "has broken decomposed plugins?"
 What's broken?  There are no links or anything.

> the IP address allocation model has changed. So someone prior to the patch
> could create a network and expect addresses A, B and C to be allocated in

Nothing has ever been documented indicating that you can expect this.
People just noticed a pattern and assumed they could count on it.
Anyone depending on this has been depending on an implementation
detail.  This has come up before [1].  I think we need flexibility to
allocate IPs as we need to to scale well.  I don't think we should be
restricted by defacto patterns in IP allocation that people have come
to depend on.

Any real world use should always take in to account the fact there
there may be other users of the system trying to get IP allocations in
parallel.  To them, the expected behavior doesn't change: they could
get any address from a window of the next few available addresses.
So, the problem here must be in tests running in a contrived setup
making too many assumptions.

> that order. Now random addresses will be allocated.

After nearly 3 years of dealing with DB contention around IP
allocation, this is the only way that we've been able to come up with
to finally relieve it.  When IPAM gets busy, there is a lot of racing
to get that next IP address which results in contention between worker
processes.  Allocating from a random window relieves it considerably.

> I think that this requires some discussion and we should consider reverting
> the patch. Maybe I am missing something here but this may break people who
> are working according the existing outputs of Neutron according to existing
> behavior (which may have been wrong from the start).

Some discussion was had [2][3] leading up to this change.  I didn't
think we needed broader discussion because we've already established
that IP allocation is an implementation detail [1].  The only contract
in place for IP allocation is that an IP address will be allocated
from within the allocation_pools defined on the subnet if available.

I am against reverting this patch as I have stated on the review to
revert it [4].

Carl

[1] https://review.openstack.org/#/c/58017/17
[2] 
http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2016-03-11.log.html#t2016-03-11T17:04:57
[3] https://bugs.launchpad.net/neutron/+bug/1543094/comments/7
[4] https://review.openstack.org/#/c/328342/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [craton] agenda for Monday's team meeting

2016-06-10 Thread sean roberts
I'd like to resync what our teams has been working on so we can create a
common agenda


-- 
~sean
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [qa] Tempest unstable interfaces in plugins

2016-06-10 Thread Andrea Frittoli
Dear all,

I'm working on making the client manager in Tempest a stable interface, so
that in future it may be used safely by plugins to easily gain access
service clients [0].

This work inevitably involves changing the current client manager
(unstable) interface.
Several tempest plugins in OpenStack already consume that interface (namely
the manager.Manager class) [1], so my work is likely to break them.

I would ask the people maintaining the plugins to be careful about using
unstable interfaces, as they are likely to change, especially since we're
working on converting them to stable.

If you maintain a plugin (in OpenStack or outside of OpenStack) that is
likely to be affected by my work, please keep an eye on my gerrit review
[0], leave a comment there or ping me on IRC (andreaf), I would very much
like to make sure the transition is as painless as possible for everyone.

andrea

[0] https://review.openstack.org/#/c/326683/
[1]
http://codesearch.openstack.org/?q=from%20tempest%20import%20manager=nope==

IRC: andreaf
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [magnum-ui][magnum] Proposed Core addition, and removal notice

2016-06-10 Thread David Lyle
+1 on my account, I've not had time to contribute.

David

On Fri, Jun 10, 2016 at 8:12 AM, Hongbin Lu  wrote:
> +1
>
>> -Original Message-
>> From: Shuu Mutou [mailto:shu-mu...@rf.jp.nec.com]
>> Sent: June-10-16 5:33 AM
>> To: openstack-dev@lists.openstack.org
>> Cc: Haruhiko Katou
>> Subject: [openstack-dev] [magnum-ui][magnum] Proposed Core addition,
>> and removal notice
>>
>> Hi team,
>>
>> I propose the following changes to the magnum-ui core group.
>>
>> + Thai Tran
>>   http://stackalytics.com/report/contribution/magnum-ui/90
>>   I'm so happy to propose Thai as a core reviewer.
>>   His reviews have been extremely valuable for us.
>>   And he is active Horizon core member.
>>   I believe his help will lead us to the correct future.
>>
>> - David Lyle
>>
>> http://stackalytics.com/?metric=marks_type=openstack=al
>> l=magnum-ui_id=david-lyle
>>   No activities for Magnum-UI since Mitaka cycle.
>>
>> - Harsh Shah
>>   http://stackalytics.com/report/users/hshah
>>   No activities for OpenStack in this year.
>>
>> - Ritesh
>>   http://stackalytics.com/report/users/rsritesh
>>   No activities for OpenStack in this year.
>>
>> Please respond with your +1 votes to approve this change or -1 votes to
>> oppose.
>>
>> Thanks,
>> Shu
>>
>>
>> ___
>> ___
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-
>> requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port isActive

2016-06-10 Thread Neil Jerram
Yes, that all makes sense - thanks for explaining.

Neil


On Fri, Jun 10, 2016 at 3:50 PM Mohammad Banikazemi  wrote:

> Hi Neil,
>
> Currently, when a docker libnetwork "join" operation in Kuryr is returned,
> it is not guaranteed that the network connectivity has been established.
> There are containers that check for network connectivity as the first thing
> they do when they come up and under heavy load some notice there is no
> connectivity and simply bail out. I am trying to deal with such a use case,
>
> Thanks for pointing out that option 2 won't work for you. I think
> Salvatore also alluded to that in his response. What you are suggesting
> with pinging the container from the appropriate namespace may be worth a
> try but then there may be containers that do not allow ingress traffic
> while they are up and happy. So short of what Salvatore suggested in his
> earlier email (and I am not sure if that can be done without additions to
> Neutron), we are left with option 1.
>
> Keep in mind that users can choose not to enable the blocking option and
> things will be as they are right now. Would that be reasonable?
>
> Best,
>
> Mohammad
>
> [image: Inactive hide details for Neil Jerram ---06/10/2016 09:25:59
> AM---Hi Mohammad, Why is the blocking needed? Is it to report som]Neil
> Jerram ---06/10/2016 09:25:59 AM---Hi Mohammad, Why is the blocking needed?
> Is it to report some kind of status back to
>
> From: Neil Jerram 
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: 06/10/2016 09:25 AM
>
>
> Subject: Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port
> is Active
> --
>
>
>
> Hi Mohammad,
>
> Why is the blocking needed?  Is it to report some kind of status back to
> Docker/Kubernetes, or to allow some follow-on action to happen?
>
> When using networking-calico as the driver, I think that only option (1)
> would work, out of the options you've suggested below.  (3) doesn't work,
> as you say, because Calico doesn't involve an L2 agent.  Also Calico
> doesn't use the RPC message queue for reporting port status, because we've
> found that that message queue is in itself a scalability bottleneck.
>
> I guess another option would be for the using system to determine for
> itself when the port appears to be working, e.g. by the host pinging the
> container/pod's IP address.
>
> Regards,
> Neil
>
>
> On Wed, Jun 8, 2016 at 4:23 PM Mohammad Banikazemi <*m...@us.ibm.com*
> > wrote:
>
>
>For the Kuryr project, in order to support blocking until vifs are
>plugged in (that is adding config options similar to the following options
>define in Nova: vif_plugging_is_fatal and vif_plugging_timeout), we need to
>detect that the Neutron plugin being used is done with plugging a given 
> vif.
>
>Here are a few options:
>
>1- The simplest approach seems to be polling for the status of the
>Neutron port to become Active. (This may lead to scalability issues but
>short of having a specific goal for scalability, it is not clear that will
>be the case.)
>2- Alternatively, We could subscribe to the message queue and wait for
>such a port update event.
>3- It was also suggested that we could use l2 agent extension to
>detect such an event but that seems to limit us to certain Neutron plugins
>and therefore not acceptable.
>
>I was wondering if there are other and better options.
>
>Best,
>
>Mohammad
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe:
>*openstack-dev-requ...@lists.openstack.org?subject:unsubscribe*
>
>
>
>*http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev*
>
>__
>
>
>
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe:
>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>
>
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally] "Failed to create the requested number of tenants" error

2016-06-10 Thread Nate Johnston
Running 'rally deployment check' appears to come up with good results:

> keystone endpoints are valid and following services are available:
> +-++---+
> | services| type   | status|
> +-++---+
> | __unknown__ | object-store2  | Available |
> | __unknown__ | s3_cluster2| Available |
> | ceilometer  | metering   | Available |
> | cinder  | volume | Available |
> | cloud   | cloudformation | Available |
> | glance  | image  | Available |
> | heat| orchestration  | Available |
> | keystone| identity   | Available |
> | neutron | network| Available |
> | nova| compute| Available |
> | s3  | s3 | Available |
> | swift   | object-store   | Available |
> +-++---+

Running `openstack network list` as you said appears to give good
results:

> [root@osrally-wc-1d ~]# openstack network list | wc -l
> 62
> [root@osrally-wc-1d ~]# openstack network list | tail -4
> | ed4e4ab4-0b3d-447a-8333-e6020221391f | sample_network_5_updated | 
> 6496fffb-47d0-4a0a-8a52-5b12ac5f15fc  
>|
> | f7d26120-057f-4dd5-a5b3-a684c4ce3350 | WayneNework  | 
> b0234e85-6e19-4f2b-ac6a-aac875ed445f  
>|
> | fd569510-3306-4b41-b97a-c4d337881128 | private-test-net | 
> f3aa1c34-d08a-41ea-87cf-019e87805a2e  
>|
> +--+--+--+

Looking at `rally deployment config | grep auth_url` shows the correct
value for the auth URL, which is the centralized keystone service.

Thanks,

--N.

On Fri, Jun 10, 2016 at 05:42:47PM +0300, Aleksandr Maretskiy wrote:
> Nate,
> 
> please try to make this simple check to make sure that everything is set up
> properly:
> 
> 1) command "rally deployment check" should print an ascii-table with a list
> of services available
> 2) load rally auto-generated openrc file and run some OpenStack CLI command,
> for example:
> 
>   $ . ~/.rally/openrc
>   $ openstack network list   # does this work as expected?
> 
> Also, make sure that value of "auth_url" in Rally deployment configuration
> (this can be obtained via command "rally deployment config") is correct.
> Please use correct deployment configuration in opposite to envvars like
> OS_AUTH_URL while using Rally
> 
> 
> On Fri, Jun 10, 2016 at 5:25 PM, Nate Johnston 
> wrote:
> 
> > Boris,
> >
> > We use a common Keystone across all of our production environments; I
> > was running this against a new deployment we are working on making
> > production-ready, so I had specified OS_AUTH_URL to be the common
> > keystone.  There is no keystone deployed in this datacenter.
> >
> > Is there a specific way I need to tweak Rally for that kind of setup?
> >
> > Thanks,
> >
> > --N.
> >
> > P.S. Sending you the catalog under separate cover.
> >
> > Thu, Jun 09, 2016 at 10:15:09PM -0700, Boris Pavlovic wrote:
> > > Nate,
> > >
> > > This looks quite strange. Could you share the information from keystone
> > > catalog?
> > >
> > > Seems like you didn't setup admin endpoint for keystone in that region.
> > >
> > > Best regards,
> > > Boris Pavlovic
> > >
> > > On Thu, Jun 9, 2016 at 12:41 PM, Nate Johnston 
> > > wrote:
> > >
> > > > Rally folks,
> > > >
> > > > I am working with an engineer to get him up to speed on Rally on a new
> > > > development.  He is trying out running a few tests from the samples
> > > > directory, like samples/tasks/scenarios/nova/list-hypervisors.yaml -
> > but
> > > > he keeps getting the error "Completed: Exit context: `users`\nTask
> > > > config is invalid: `Unable to setup context 'users': 'Failed to create
> > > > the requested number of tenants.'`"
> > > >
> > > > This is against an Icehouse environment with Mitaka Rally; When I run
> > > > Rally with debug logging I see:
> > > >
> > > > 2016-06-08 18:59:24.692 11197 ERROR rally.common.broker
> > EndpointNotFound:
> > > > admin endpoint for identity service in  region not found
> > > >
> > > > However I note that $OS_AUTH_URL is set in the Rally deployment... see
> > > > http://paste.openstack.org/show/509002/ for the full log.
> > > >
> > > > Any ideas you could give me would be much appreciated.  Thanks!
> > > >
> > > > --N.
> > > >
> > > >
> > > >
> > > >
> > __
> > > > OpenStack Development Mailing List (not for usage questions)
> > > > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> 

[openstack-dev] [StoryBoard] StoryBoard Bug Squash!

2016-06-10 Thread Zara Zaimeche

Hi all!

We're running a bugsquash for bugs in StoryBoard and the StoryBoard 
webclient, on the 22nd and 23rd of June. It'll start and finish around 
11:00 UTC-- though of course every day is a StoryBoard bugsquash day 
*really*, so that's just the timeframe where StoryBoard people will 
sometimes be active in #openstack-sprint on freenode. We'll also be 
around in #storyboard, which is where we live the rest of the time. If 
you're interested in helping out with the bugs, then it's a good time to 
start! We're going to try to be more vigilant actually tagging 
low-hanging-fruit as low-hanging-fruit in the next couple of weeks, so 
that it's easy for people to see what needs work. :) Feel free to join 
#storyboard if you want to squash some bugs. Or if you have questions. 
Or if you just want to keep us company.


Best Wishes,

Zara (and SotK probably also wishes you all well, but he hasn't seen 
this email, so I won't put his name on it, except in parentheses like these)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Dmitry Tantsur

On 06/10/2016 02:16 PM, Jim Rollenhagen wrote:

On Thu, Jun 09, 2016 at 03:21:05PM -0700, Jay Faulkner wrote:

A quick update:

The devstack-gate patch is currently merging.

There was some discussion about whether or not the Ironic grenade job should
be in the check pipeline (even as -nv) for grenade, so I split that patch
into two pieces so the less controversial part (adding the grenade-nv job to
Ironic's check pipeline) could merge more easily.

https://review.openstack.org/#/c/319336/ - project-config
Make grenade-dsvm-ironic non voting (in the check queue for Ironic only)

https://review.openstack.org/#/c/327985/ - project-config
Make grenade-dsvm-ironic non voting (in the check queue for grenade)

Getting working upgrade testing will be a huge milestone for Ironic. Thanks
to those who have already helped us make progress and those who will help us
land these and see it at work.


The first of these is merged, which gets us most of what we needed.
Thanks everyone for your hard work on this!


Let me also say thank you to all the folks who spent huge amount of time 
testing, hacking, fixing and reviewing stuff for us to pass this 
important milestone. Thanks!




/me rechecks some things waiting for this testing :D

// jim



Thanks in advance,
Jay Faulkner
OSIC

On 6/9/16 8:28 AM, Jim Rollenhagen wrote:

Hi friends,

We're two patches away from having grenade passing in our check queue!
This is a huge step forward for us, many thanks go to the numerous folks
that have worked on or helped somehow with this.

I'd love to push this across the line today as it's less than 10 lines
of changes between the two, and we have a bunch of work nearly done that
we'd like upgrade testing running against before merging.

So we need infra cores' help here.

https://review.openstack.org/#/c/316662/ - devstack-gate
Allow to pass OS_TEST_TIMEOUT for grenade job
1 line addition with an sdague +2.

https://review.openstack.org/#/c/319336/ - project-config
Make grenade-dsvm-ironic non voting (in the check queue)
+7,-1 with an AJaeger +2.

Thanks in advance. :)

// jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [neutron] [classifier] Requesting feedback on typed classifications model

2016-06-10 Thread Duarte Cardoso, Igor
Hi community and common classifier team,

In anticipation to the next common classifier meeting, I've added a meeting 
agenda item to the next meeting's agenda [1] about the proposed "typed 
classifications" model for the new common classifier.

The typed classifications model is described in [2, comment #26] and 
essentially separates classification matches into different classification 
types, instead of grouping all kinds of different classifications into a single 
monolithic structure such as [3].

This kind of model is actually not new, as neutron-classifier was already 
following such a path [4].
Can someone give feedback on the described model and how neutron-classifier is 
planned to be reused for the new effort?

[1] https://wiki.openstack.org/wiki/Neutron/CommonFlowClassifier
[2] https://bugs.launchpad.net/neutron/+bug/1476527
[3] https://wiki.openstack.org/w/images/c/c8/Neutron_Common_Classifier.png
[4] 
https://github.com/openstack/neutron-classifier/blob/10b2eb3127f4809e52e3cf1627c34228bca80101/neutron_classifier/common/constants.py#L17

Best regards,
Igor.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][glance][qa] - Nova glance v2 work complete

2016-06-10 Thread Mikhail Fedosin
Hooray :) After a long period of time it's done. Thanks all who helped us
there!

Best regards,
Mike Fedosin

On Fri, Jun 10, 2016 at 3:27 PM, Kairat Kushaev 
wrote:

> \0/
> That's awesome.
> Big thanks to mfedosin and sudipto for driving this work.
>
> Best regards,
> Kairat Kushaev
>
> On Fri, Jun 10, 2016 at 2:52 PM, Monty Taylor 
> wrote:
>
>> On 06/10/2016 01:19 PM, Sean Dague wrote:
>> > On 06/07/2016 04:55 PM, Matt Riedemann wrote:
>> >> I tested the glance v2 stack (glance v1 disabled) using a devstack
>> >> change here:
>> >>
>> >> https://review.openstack.org/#/c/325322/
>> >>
>> >> Now that the changes are merged up through the base nova image proxy
>> and
>> >> the libvirt driver, and we just have hyper-v/xen driver changes for
>> that
>> >> series, we should look at gating on this configuration.
>> >>
>> >> I was originally thinking about adding a new job for this, but it's
>> >> probably better if we just change one of the existing integrated gate
>> >> jobs, like gate-tempest-dsvm-full or gate-tempest-dsvm-neutron-full.
>> >>
>> >> Does anyone have an issue with that? Glance v1 is deprecated and the
>> >> configuration option added to nova (use_glance_v1) defaults to True for
>> >> compat but is deprecated, and the Nova team plans to drop it's v1 proxy
>> >> code in Ocata. So it seems like changing config to use v2 in the gate
>> >> jobs should be a non-issue. We'd want to keep at least one integrated
>> >> gate job using glance v1 to make sure we don't regress anything there
>> in
>> >> Newton.
>> >
>> > use_glance_v1=False has now been merged as the default, so all jobs are
>> > now using glance v2 for the Nova <=> Glance communication -
>> > https://review.openstack.org/#/c/321551/
>> >
>> > Thanks to Mike and Sudipta for pushing this to completion.
>>
>> Congrats everybody!!!
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port isActive

2016-06-10 Thread Mohammad Banikazemi

Hi Neil,

Currently, when a docker libnetwork "join" operation in Kuryr is returned,
it is not guaranteed that the network connectivity has been established.
There are containers that check for network connectivity as the first thing
they do when they come up and under heavy load some notice there is no
connectivity and simply bail out. I am trying to deal with such a use case,

Thanks for pointing out that option 2 won't work for you. I think Salvatore
also alluded to that in his response. What you are suggesting with pinging
the container from the appropriate namespace may be worth a try but then
there may be containers that do not allow ingress traffic while they are up
and happy. So short of what Salvatore suggested in his earlier email (and I
am not sure if that can be done without additions to Neutron), we are left
with option 1.

Keep in mind that users can choose not to enable the blocking option and
things will be as they are right now. Would that be reasonable?

Best,

Mohammad



From:   Neil Jerram 
To: "OpenStack Development Mailing List (not for usage questions)"

Date:   06/10/2016 09:25 AM
Subject:Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron
Port is Active



Hi Mohammad,

Why is the blocking needed?  Is it to report some kind of status back to
Docker/Kubernetes, or to allow some follow-on action to happen?

When using networking-calico as the driver, I think that only option (1)
would work, out of the options you've suggested below.  (3) doesn't work,
as you say, because Calico doesn't involve an L2 agent.  Also Calico
doesn't use the RPC message queue for reporting port status, because we've
found that that message queue is in itself a scalability bottleneck.

I guess another option would be for the using system to determine for
itself when the port appears to be working, e.g. by the host pinging the
container/pod's IP address.

Regards,
    Neil


On Wed, Jun 8, 2016 at 4:23 PM Mohammad Banikazemi  wrote:
  For the Kuryr project, in order to support blocking until vifs are
  plugged in (that is adding config options similar to the following
  options define in Nova: vif_plugging_is_fatal and vif_plugging_timeout),
  we need to detect that the Neutron plugin being used is done with
  plugging a given vif.

  Here are a few options:

  1- The simplest approach seems to be polling for the status of the
  Neutron port to become Active. (This may lead to scalability issues but
  short of having a specific goal for scalability, it is not clear that
  will be the case.)
  2- Alternatively, We could subscribe to the message queue and wait for
  such a port update event.
  3- It was also suggested that we could use l2 agent extension to detect
  such an event but that seems to limit us to certain Neutron plugins and
  therefore not acceptable.

  I was wondering if there are other and better options.

  Best,

  Mohammad
  __

  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
  __

  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [rally] "Failed to create the requested number of tenants" error

2016-06-10 Thread Aleksandr Maretskiy
Nate,

please try to make this simple check to make sure that everything is set up
properly:

1) command "rally deployment check" should print an ascii-table with a list
of services available
2) load rally auto-generated openrc file and run some OpenStack CLI command,
for example:

  $ . ~/.rally/openrc
  $ openstack network list   # does this work as expected?

Also, make sure that value of "auth_url" in Rally deployment configuration
(this can be obtained via command "rally deployment config") is correct.
Please use correct deployment configuration in opposite to envvars like
OS_AUTH_URL while using Rally


On Fri, Jun 10, 2016 at 5:25 PM, Nate Johnston 
wrote:

> Boris,
>
> We use a common Keystone across all of our production environments; I
> was running this against a new deployment we are working on making
> production-ready, so I had specified OS_AUTH_URL to be the common
> keystone.  There is no keystone deployed in this datacenter.
>
> Is there a specific way I need to tweak Rally for that kind of setup?
>
> Thanks,
>
> --N.
>
> P.S. Sending you the catalog under separate cover.
>
> Thu, Jun 09, 2016 at 10:15:09PM -0700, Boris Pavlovic wrote:
> > Nate,
> >
> > This looks quite strange. Could you share the information from keystone
> > catalog?
> >
> > Seems like you didn't setup admin endpoint for keystone in that region.
> >
> > Best regards,
> > Boris Pavlovic
> >
> > On Thu, Jun 9, 2016 at 12:41 PM, Nate Johnston 
> > wrote:
> >
> > > Rally folks,
> > >
> > > I am working with an engineer to get him up to speed on Rally on a new
> > > development.  He is trying out running a few tests from the samples
> > > directory, like samples/tasks/scenarios/nova/list-hypervisors.yaml -
> but
> > > he keeps getting the error "Completed: Exit context: `users`\nTask
> > > config is invalid: `Unable to setup context 'users': 'Failed to create
> > > the requested number of tenants.'`"
> > >
> > > This is against an Icehouse environment with Mitaka Rally; When I run
> > > Rally with debug logging I see:
> > >
> > > 2016-06-08 18:59:24.692 11197 ERROR rally.common.broker
> EndpointNotFound:
> > > admin endpoint for identity service in  region not found
> > >
> > > However I note that $OS_AUTH_URL is set in the Rally deployment... see
> > > http://paste.openstack.org/show/509002/ for the full log.
> > >
> > > Any ideas you could give me would be much appreciated.  Thanks!
> > >
> > > --N.
> > >
> > >
> > >
> > >
> __
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
>
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [stable][all] Tagging kilo-eol for "the world"

2016-06-10 Thread Boden Russell
On 6/2/16 4:31 AM, Tony Breeds wrote:
> Any projects that will be EOL'd will need all open reviews abandoned before it
> can be processed. 

openstack/vmware-nsx kilo patches have been abandoned in preparation for
the EOL.

Thanks

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron][SFC]

2016-06-10 Thread Alioune
Hi Mohan,
Even if I clone the master branch of networking-sfc project,I get the
following errir when creating flow-classifier, therefore I do precise the
 logical-source-port.

2016-06-10 05:34:05.693 10799 ERROR neutron.api.v2.resource DBError:
(pymysql.err.IntegrityError) (1048, u"Column 'logical_source_port' cannot
be null")

I'm trying the example in [1]

Here is a "ovs-ofctl dump-flows" on br-int ofter creating port-chain, I
expected to see vxlan or gre tunnel encapsulation entries as explained in
[1], may I know why there is no tunnel entry in br-int ?

sudo ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
 cookie=0x90444f3c8fabcbe0, duration=2840.983s, table=0, n_packets=0,
n_bytes=0, idle_age=2840, priority=10,icmp6,in_port=16,icmp_type=136
actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2837.039s, table=0, n_packets=0,
n_bytes=0, idle_age=2837, priority=10,icmp6,in_port=17,icmp_type=136
actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2831.688s, table=0, n_packets=0,
n_bytes=0, idle_age=2831, priority=10,icmp6,in_port=19,icmp_type=136
actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2831.038s, table=0, n_packets=0,
n_bytes=0, idle_age=2831, priority=10,icmp6,in_port=18,icmp_type=136
actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2801.555s, table=0, n_packets=0,
n_bytes=0, idle_age=2801, priority=10,icmp6,in_port=20,icmp_type=136
actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2840.605s, table=0, n_packets=8,
n_bytes=336, idle_age=2591, priority=10,arp,in_port=16 actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2836.759s, table=0, n_packets=0,
n_bytes=0, idle_age=2836, priority=10,arp,in_port=17 actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2831.485s, table=0, n_packets=0,
n_bytes=0, idle_age=2831, priority=10,arp,in_port=19 actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2830.816s, table=0, n_packets=21,
n_bytes=882, idle_age=1605, priority=10,arp,in_port=18 actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=2801.309s, table=0, n_packets=10,
n_bytes=420, idle_age=545, priority=10,arp,in_port=20 actions=resubmit(,24)
 cookie=0x90444f3c8fabcbe0, duration=15755.073s, table=0, n_packets=3241,
n_bytes=366555, idle_age=545, priority=0 actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=15754.687s, table=23, n_packets=0,
n_bytes=0, idle_age=15754, priority=0 actions=drop
 cookie=0x90444f3c8fabcbe0, duration=2841.201s, table=24, n_packets=0,
n_bytes=0, idle_age=2841,
priority=2,icmp6,in_port=16,icmp_type=136,nd_target=fe80::f816:3eff:fe2d:c29d
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2837.177s, table=24, n_packets=0,
n_bytes=0, idle_age=2837,
priority=2,icmp6,in_port=17,icmp_type=136,nd_target=fe80::f816:3eff:fee0:f8ca
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2831.794s, table=24, n_packets=0,
n_bytes=0, idle_age=2831,
priority=2,icmp6,in_port=19,icmp_type=136,nd_target=fe80::f816:3eff:fe86:a668
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2831.150s, table=24, n_packets=0,
n_bytes=0, idle_age=2831,
priority=2,icmp6,in_port=18,icmp_type=136,nd_target=fe80::f816:3eff:feb4:965f
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2801.675s, table=24, n_packets=0,
n_bytes=0, idle_age=2801,
priority=2,icmp6,in_port=20,icmp_type=136,nd_target=fe80::f816:3eff:fe5a:3097
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2840.794s, table=24, n_packets=8,
n_bytes=336, idle_age=2591, priority=2,arp,in_port=16,arp_spa=55.55.55.3
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2836.901s, table=24, n_packets=0,
n_bytes=0, idle_age=2836, priority=2,arp,in_port=17,arp_spa=55.55.55.4
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2831.587s, table=24, n_packets=0,
n_bytes=0, idle_age=2831, priority=2,arp,in_port=19,arp_spa=55.55.55.6
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2830.933s, table=24, n_packets=21,
n_bytes=882, idle_age=1605, priority=2,arp,in_port=18,arp_spa=55.55.55.5
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=2801.431s, table=24, n_packets=10,
n_bytes=420, idle_age=545, priority=2,arp,in_port=20,arp_spa=55.55.55.8
actions=NORMAL
 cookie=0x90444f3c8fabcbe0, duration=15754.273s, table=24, n_packets=0,
n_bytes=0, idle_age=15754, priority=0 actions=drop


is there a link that explains how pipelines are created by SFC in br-int to
compare with my flows entries ?

Does the flow-classifier referer to floating IPs or tenant network IPs to
do the classification ?

Regards,

[1]
http://docs.openstack.org/developer/networking-sfc/system_design%20and_workflow.html

On 9 June 2016 at 18:59, Henry Fourie  wrote:

> Alioune,
>
>The logical-source-port refers to a Neutron port of the VM that
>
> originates the traffic that is to be processed by the port-chain.
>
> -Louis
>
>
>
> *From:* Alioune [mailto:baliou...@gmail.com]
> *Sent:* Thursday, June 09, 2016 6:50 AM
> *To:* Mohan Kumar
> *Cc:* OpenStack Development Mailing List (not 

Re: [openstack-dev] [rally] "Failed to create the requested number of tenants" error

2016-06-10 Thread Nate Johnston
Boris,

We use a common Keystone across all of our production environments; I
was running this against a new deployment we are working on making
production-ready, so I had specified OS_AUTH_URL to be the common
keystone.  There is no keystone deployed in this datacenter.

Is there a specific way I need to tweak Rally for that kind of setup?

Thanks,

--N.

P.S. Sending you the catalog under separate cover.

Thu, Jun 09, 2016 at 10:15:09PM -0700, Boris Pavlovic wrote:
> Nate,
> 
> This looks quite strange. Could you share the information from keystone
> catalog?
> 
> Seems like you didn't setup admin endpoint for keystone in that region.
> 
> Best regards,
> Boris Pavlovic
> 
> On Thu, Jun 9, 2016 at 12:41 PM, Nate Johnston 
> wrote:
> 
> > Rally folks,
> >
> > I am working with an engineer to get him up to speed on Rally on a new
> > development.  He is trying out running a few tests from the samples
> > directory, like samples/tasks/scenarios/nova/list-hypervisors.yaml - but
> > he keeps getting the error "Completed: Exit context: `users`\nTask
> > config is invalid: `Unable to setup context 'users': 'Failed to create
> > the requested number of tenants.'`"
> >
> > This is against an Icehouse environment with Mitaka Rally; When I run
> > Rally with debug logging I see:
> >
> > 2016-06-08 18:59:24.692 11197 ERROR rally.common.broker EndpointNotFound:
> > admin endpoint for identity service in  region not found
> >
> > However I note that $OS_AUTH_URL is set in the Rally deployment... see
> > http://paste.openstack.org/show/509002/ for the full log.
> >
> > Any ideas you could give me would be much appreciated.  Thanks!
> >
> > --N.
> >
> >
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [magnum-ui][magnum] Proposed Core addition, and removal notice

2016-06-10 Thread Hongbin Lu
+1

> -Original Message-
> From: Shuu Mutou [mailto:shu-mu...@rf.jp.nec.com]
> Sent: June-10-16 5:33 AM
> To: openstack-dev@lists.openstack.org
> Cc: Haruhiko Katou
> Subject: [openstack-dev] [magnum-ui][magnum] Proposed Core addition,
> and removal notice
> 
> Hi team,
> 
> I propose the following changes to the magnum-ui core group.
> 
> + Thai Tran
>   http://stackalytics.com/report/contribution/magnum-ui/90
>   I'm so happy to propose Thai as a core reviewer.
>   His reviews have been extremely valuable for us.
>   And he is active Horizon core member.
>   I believe his help will lead us to the correct future.
> 
> - David Lyle
> 
> http://stackalytics.com/?metric=marks_type=openstack=al
> l=magnum-ui_id=david-lyle
>   No activities for Magnum-UI since Mitaka cycle.
> 
> - Harsh Shah
>   http://stackalytics.com/report/users/hshah
>   No activities for OpenStack in this year.
> 
> - Ritesh
>   http://stackalytics.com/report/users/rsritesh
>   No activities for OpenStack in this year.
> 
> Please respond with your +1 votes to approve this change or -1 votes to
> oppose.
> 
> Thanks,
> Shu
> 
> 
> ___
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone] Changing the project name uniqueness constraint

2016-06-10 Thread Henry Nash
On further reflection, it seems to me that we can never simply enable either of 
these approaches in a single release. Even a v4.0 version of the API doesn’t 
help - since presumably a sever supporting v4 would want to be able to support 
v3.x for a signification time; and, already discussed, as soon as you allow 
multiple none-names to have the same name, you can no longer guarantee to 
support the current API.

Hence the only thing I think we can do (if we really do want to change the 
current functionality) is to do this over several releases with a typical 
deprecation cycle, e.g.

1) At release 3.7 we allow you to (optionally) specify path names for auth….but 
make no changes to the uniqueness constraints. We also change the GET 
/auth/projects to return a path name. However, you can still auth exactly the 
way we do today (since there will always only be a single project of a given 
node-name). If however, you do auth without a path (to a project that isn’t a 
top level project), we log a warning to say this is deprecated (2 cycles, 4 
cycles?)
2) If you connect with a 3.6 client, then you get the same as today for GET 
/auth/projects and cannot use a path name to auth.
3) At sometime in the future, we deprecate the “auth without a path” 
capability. We can debate as to whether this has to be a major release.

If we take this gradual approach, I would be pushing for the “relax project 
name constraints” approach…since I believe this leads to a cleaner eventual 
solution (and there is no particular advantage with the hierarchical naming 
approach) - and (until the end of the deprecation) there is no break to the 
existing API.

Henry
> On 7 Jun 2016, at 09:47, Henry Nash  wrote:
> 
> OK, so thanks for the feedback - understand the message.
> 
> However, in terms of compatibility, the one thing that concerns me about the 
> hierarchical naming approach is that even with microversioing, we might still 
> surprise a client. An unmodified client (i.e. doesn’t understand 3.7) would 
> still see a change in the data being returned (the project names have 
> suddenly become full path names). We have to return this even if they don’t 
> ask for 3.7, since otherwise there is no difference between this approach and 
> relaxing the project naming in terms of trying to prevent auth breakages.
> 
> In more detail:
> 
> 1) Both approaches were planned to return the path name (instead of the node 
> name) in GET /auth/projects - i.e. the API you are meant to use to find out 
> what you can scope to
> 2) Both approaches were planned to accept the path name in the auth request 
> block
> 3) The difference in hierarchical naming is the if I do a regular GET 
> /project(s) I also see the full path name as the “project name”
> 
> if we don’t do 3), then code that somehow authenticates, and then uses the 
> regular GET /project(s) calls to find a project name and then re-scopes (or 
> re-auths) to that name, will fail if the project they want is not a top level 
> project. However, the flip side is that if there is code that uses these same 
> calls to, say, display projects to the user (e.g. a home grown UI) - then it 
> might get confused until it supports 3.7 (i.e. asking for the old 
> microversion won’t help it) since all the names include the hierarchical path.
> 
> Just want to make sure we understand the implications….
> 
> Henry
> 
>> On 4 Jun 2016, at 08:34, Monty Taylor > > wrote:
>> 
>> On 06/04/2016 01:53 AM, Morgan Fainberg wrote:
>>> 
>>> On Jun 3, 2016 12:42, "Lance Bragstad" >> 
>>> >> wrote:
 
 
 
 On Fri, Jun 3, 2016 at 11:20 AM, Henry Nash 
>>> >> wrote:
> 
> 
>> On 3 Jun 2016, at 16:38, Lance Bragstad > 
>>> >> wrote:
>> 
>> 
>> 
>> On Fri, Jun 3, 2016 at 3:20 AM, Henry Nash > 
>>> >> wrote:
>>> 
>>> 
 On 3 Jun 2016, at 01:22, Adam Young 
>>> >> wrote:
 
 On 06/02/2016 07:22 PM, Henry Nash wrote:
> 
> Hi
> 
> As you know, I have been working on specs that change the way we
>>> handle the uniqueness of project names in Newton. The goal of this is to
>>> better support project hierarchies, which as they stand today are
>>> restrictive in that all project names within a domain must be unique,
>>> irrespective of where in the hierarchy that projects sits (unlike, say,
>>> the unix directory 

Re: [openstack-dev] Quesion about Openstack Containers and Magnum

2016-06-10 Thread Spyros Trigazis
Hi Wally.

You can follow these instructions [1] to install from source
code (use the stable/mitaka branch when you'll clone).

Although, this guide is under the developer url, it's built for
operators. Keep in mind that you need Neutron/LBaaS V1.

Additionally there is this [2] puppet module.

Please share your experience.

Cheers,
Spyros

[1]
http://docs.openstack.org/developer/magnum/install-guide-from-source.html
[2] https://github.com/openstack/puppet-magnum

On 10 June 2016 at 06:26, zhihao wang  wrote:

> Dear Openstack Dev Members:
>
> I would like to install the Magnum on OpenStack to manage Docker
> Containers.
> I have a openstack Liberty production setup. one controller node, and a
> few compute nodes.
>
> I am wondering how can I install Openstack Magnum on OpenStack Liberty on
> distributed production environment (1 controller node and some compute
> nodes)?
>
> I know I can install Magnum using desstack, but I dont want the developer
> version,
>
> Is there a way/guide to install it on production environment?
>
> Thanks
> Wally
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Kuryr] [Neutron] Waiting until Neutron Port is Active

2016-06-10 Thread Neil Jerram
Hi Mohammad,

Why is the blocking needed?  Is it to report some kind of status back to
Docker/Kubernetes, or to allow some follow-on action to happen?

When using networking-calico as the driver, I think that only option (1)
would work, out of the options you've suggested below.  (3) doesn't work,
as you say, because Calico doesn't involve an L2 agent.  Also Calico
doesn't use the RPC message queue for reporting port status, because we've
found that that message queue is in itself a scalability bottleneck.

I guess another option would be for the using system to determine for
itself when the port appears to be working, e.g. by the host pinging the
container/pod's IP address.

Regards,
Neil


On Wed, Jun 8, 2016 at 4:23 PM Mohammad Banikazemi  wrote:

> For the Kuryr project, in order to support blocking until vifs are plugged
> in (that is adding config options similar to the following options define
> in Nova: vif_plugging_is_fatal and vif_plugging_timeout), we need to detect
> that the Neutron plugin being used is done with plugging a given vif.
>
> Here are a few options:
>
> 1- The simplest approach seems to be polling for the status of the Neutron
> port to become Active. (This may lead to scalability issues but short of
> having a specific goal for scalability, it is not clear that will be the
> case.)
> 2- Alternatively, We could subscribe to the message queue and wait for
> such a port update event.
> 3- It was also suggested that we could use l2 agent extension to detect
> such an event but that seems to limit us to certain Neutron plugins and
> therefore not acceptable.
>
> I was wondering if there are other and better options.
>
> Best,
>
> Mohammad
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

2016-06-10 Thread Paul Michali
Actually, I had menm_page_size set to "large" and not "1024". However, it
seemed like it was using 1024 pages per (small VM creation). Is there
possibly some issue with large not using one of the supported values? I
would have guessed it would have chosen 2M or 1G for the size.

Any thoughts?

PCM

On Fri, Jun 10, 2016 at 9:05 AM Paul Michali  wrote:

> Thanks Daniel and Chris! I think that was the problem, I had configured
> Nova flavor with a mem_page_size of 1024, and it should have been one of
> the supported values.
>
> I'll go through and check things out one more time, but I think that is
> the problem. I still need to figure out what is going on with the neutron
> port not being released - we have another person in my group who has seen
> the same issue.
>
> Regards,
>
> PCM
>
> On Fri, Jun 10, 2016 at 4:41 AM Daniel P. Berrange 
> wrote:
>
>> On Thu, Jun 09, 2016 at 12:35:06PM -0600, Chris Friesen wrote:
>> > On 06/09/2016 05:15 AM, Paul Michali wrote:
>> > > 1) On the host, I was seeing 32768 huge pages, of 2MB size.
>> >
>> > Please check the number of huge pages _per host numa node_.
>> >
>> > > 2) I changed mem_page_size from 1024 to 2048 in the flavor, and then
>> when VMs
>> > > were created, they were being evenly assigned to the two NUMA nodes.
>> Each using
>> > > 1024 huge pages. At this point I could create more than half, but
>> when there
>> > > were 1945 pages left, it failed to create a VM. Did it fail because
>> the
>> > > mem_page_size was 2048 and the available pages were 1945, even though
>> we were
>> > > only requesting 1024 pages?
>> >
>> > I do not think that "1024" is a valid page size (at least for x86).
>>
>> Correct, 4k, 2M and 1GB are valid page sizes.
>>
>> > Valid mem_page_size values are determined by the host CPU.  You do not
>> need
>> > a larger page size for flavors with larger memory sizes.
>>
>> Though note that page sizes should be a multiple of favour mem size
>> unless you want to waste memory. eg if you have a flavour with 750MB
>> RAM, then you probably don't want to use 1GB pages as it waste 250MB
>>
>> Regards,
>> Daniel
>> --
>> |: http://berrange.com  -o-
>> http://www.flickr.com/photos/dberrange/ :|
>> |: http://libvirt.org  -o-
>> http://virt-manager.org :|
>> |: http://autobuild.org   -o-
>> http://search.cpan.org/~danberr/ :|
>> |: http://entangle-photo.org   -o-
>> http://live.gnome.org/gtk-vnc :|
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

2016-06-10 Thread Paul Michali
Thanks Daniel and Chris! I think that was the problem, I had configured
Nova flavor with a mem_page_size of 1024, and it should have been one of
the supported values.

I'll go through and check things out one more time, but I think that is the
problem. I still need to figure out what is going on with the neutron port
not being released - we have another person in my group who has seen the
same issue.

Regards,

PCM

On Fri, Jun 10, 2016 at 4:41 AM Daniel P. Berrange 
wrote:

> On Thu, Jun 09, 2016 at 12:35:06PM -0600, Chris Friesen wrote:
> > On 06/09/2016 05:15 AM, Paul Michali wrote:
> > > 1) On the host, I was seeing 32768 huge pages, of 2MB size.
> >
> > Please check the number of huge pages _per host numa node_.
> >
> > > 2) I changed mem_page_size from 1024 to 2048 in the flavor, and then
> when VMs
> > > were created, they were being evenly assigned to the two NUMA nodes.
> Each using
> > > 1024 huge pages. At this point I could create more than half, but when
> there
> > > were 1945 pages left, it failed to create a VM. Did it fail because the
> > > mem_page_size was 2048 and the available pages were 1945, even though
> we were
> > > only requesting 1024 pages?
> >
> > I do not think that "1024" is a valid page size (at least for x86).
>
> Correct, 4k, 2M and 1GB are valid page sizes.
>
> > Valid mem_page_size values are determined by the host CPU.  You do not
> need
> > a larger page size for flavors with larger memory sizes.
>
> Though note that page sizes should be a multiple of favour mem size
> unless you want to waste memory. eg if you have a flavour with 750MB
> RAM, then you probably don't want to use 1GB pages as it waste 250MB
>
> Regards,
> Daniel
> --
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/
> :|
> |: http://libvirt.org  -o- http://virt-manager.org
> :|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/
> :|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc
> :|
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron][networking-ovn] OVN vs. OpenDayLight

2016-06-10 Thread Russell Bryant
On Fri, Jun 10, 2016 at 11:11 AM, Oleg Bondarev 
wrote:

> Hi,
>
> [1] is a doc comparing OVN and ODL for Neutron. I wrote it in March so
> some info might be stale.
> Hope this can be useful, comments are welcome!
>
> [1]
> https://docs.google.com/document/d/13K4Xpdu1IbRJDj5JTepDI4nwUkCDCu7IOnYwHkZtcMA/edit?usp=sharing
>

​Nice work, Oleg.  I'll comment on the doc for things that I know of that
have changed since March.

Thanks,

-- 
Russell Bryant​



-- 
Russell Bryant
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

2016-06-10 Thread Paul Michali
See PCM: Inline...


On Thu, Jun 9, 2016 at 11:42 AM Steve Gordon  wrote:

> - Original Message -
> > From: "Paul Michali" 
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> > Sent: Tuesday, June 7, 2016 11:00:30 AM
> > Subject: Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling
> >
> > Anyone have any thoughts on the two questions below? Namely...
> >
> > If the huge pages are 2M, we are creating a 2GB VM, have 1945 huge pages,
> > should the allocation fail (and if so why)?
>
> Were enough pages (1024) available in a single NUMA node? Which release
> are you using? There was a bug where node 0 would always be picked (and
> eventually exhausted) but that was - theoretically - fixed under
> https://bugs.launchpad.net/nova/+bug/1386236


PCM: This is on LIberty, so it sounds like the bugfix was in there.  It's
possible that there was not 1024 left, on a single NUMA node.

Regards,

PCM


>
>
> > Why do all the 2GB VMs get created on the same NUMA node, instead of
> > getting evenly assigned to each of the two NUMA nodes that are available
> on
> > the compute node (as a result, allocation fails, when 1/2 the huge pages
> > are used)? I found that increasing mem_page_size to 2048 resolves the
> > issue, but don't know why.
>
> What was the mem_page_size before it was 2048? I didn't think any smaller
> value was supported.
>
> > ANother thing I was seeing, when the VM create failed due to not enough
> > huge pages available and was in error state, I could delete the VM, but
> the
> > Neutron port was still there.  Is that correct?
> >
> > I didn't see any log messages in neutron, requesting to unbind and delete
> > the port.
> >
> > Thanks!
> >
> > PCM
> >
> > .
> >
> > On Fri, Jun 3, 2016 at 2:03 PM Paul Michali  wrote:
> >
> > > Thanks for the link Tim!
> > >
> > > Right now, I have two things I'm unsure about...
> > >
> > > One is that I had 1945 huge pages left (of size 2048k) and tried to
> create
> > > a VM with a small flavor (2GB), which should need 1024 pages, but Nova
> > > indicated that it wasn't able to find a host (and QEMU reported an
> > > allocation issue).
> > >
> > > The other is that VMs are not being evenly distributed on my two NUMA
> > > nodes, and instead, are getting created all on one NUMA node. Not sure
> if
> > > that is expected (and setting mem_page_size to 2048 is the proper way).
> > >
> > > Regards,
> > >
> > > PCM
> > >
> > >
> > > On Fri, Jun 3, 2016 at 1:21 PM Tim Bell  wrote:
> > >
> > >> The documentation at
> > >> http://docs.openstack.org/admin-guide/compute-flavors.html is
> gradually
> > >> improving. Are there areas which were not covered in your
> clarifications ?
> > >> If so, we should fix the documentation too since this is a complex
> area to
> > >> configure and good documentation is a great help.
> > >>
> > >>
> > >>
> > >> BTW, there is also an issue around how the RAM for the BIOS is
> shadowed.
> > >> I can’t find the page from a quick google but we found an imbalance
> when
> > >> we
> > >> used 2GB pages as the RAM for BIOS shadowing was done by default in
> the
> > >> memory space for only one of the NUMA spaces.
> > >>
> > >>
> > >>
> > >> Having a look at the KVM XML can also help a bit if you are debugging.
> > >>
> > >>
> > >>
> > >> Tim
> > >>
> > >>
> > >>
> > >> *From: *Paul Michali 
> > >> *Reply-To: *"OpenStack Development Mailing List (not for usage
> > >> questions)" 
> > >> *Date: *Friday 3 June 2016 at 15:18
> > >> *To: *"Daniel P. Berrange" , "OpenStack
> Development
> > >> Mailing List (not for usage questions)" <
> > >> openstack-dev@lists.openstack.org>
> > >> *Subject: *Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling
> > >>
> > >>
> > >>
> > >> See PCM inline...
> > >>
> > >> On Fri, Jun 3, 2016 at 8:44 AM Daniel P. Berrange <
> berra...@redhat.com>
> > >> wrote:
> > >>
> > >> On Fri, Jun 03, 2016 at 12:32:17PM +, Paul Michali wrote:
> > >> > Hi!
> > >> >
> > >> > I've been playing with Liberty code a bit and had some questions
> that
> > >> I'm
> > >> > hoping Nova folks may be able to provide guidance on...
> > >> >
> > >> > If I set up a flavor with hw:mem_page_size=2048, and I'm creating
> > >> (Cirros)
> > >> > VMs with size 1024, will the scheduling use the minimum of the
> number of
> > >>
> > >> 1024 what units ? 1024 MB, or 1024 huge pages aka 2048 MB ?
> > >>
> > >>
> > >>
> > >> PCM: I was using small flavor, which is 2 GB. So that's 2048 MB and
> the
> > >> page size is 2048K, so 1024 pages? Hope I have the units right.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> > huge pages available and the size requested for the VM, or will it
> base
> > >> > scheduling only on the number of huge pages?
> > >> >
> > >> > It seems to be doing the latter, where I had 1945 huge pages free,
> and

Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

2016-06-10 Thread Paul Michali
I'll try to reproduce and collect logs for a bug report.

Thanks for the info.

PCM


On Thu, Jun 9, 2016 at 9:43 AM Matt Riedemann 
wrote:

> On 6/9/2016 6:15 AM, Paul Michali wrote:
> >
> >
> > On Wed, Jun 8, 2016 at 11:21 PM Chris Friesen
> > >
> wrote:
> >
> > On 06/03/2016 12:03 PM, Paul Michali wrote:
> > > Thanks for the link Tim!
> > >
> > > Right now, I have two things I'm unsure about...
> > >
> > > One is that I had 1945 huge pages left (of size 2048k) and tried
> > to create a VM
> > > with a small flavor (2GB), which should need 1024 pages, but Nova
> > indicated that
> > > it wasn't able to find a host (and QEMU reported an allocation
> issue).
> > >
> > > The other is that VMs are not being evenly distributed on my two
> > NUMA nodes, and
> > > instead, are getting created all on one NUMA node. Not sure if
> > that is expected
> > > (and setting mem_page_size to 2048 is the proper way).
> >
> >
> > Just in case you haven't figured out the problem...
> >
> > Have you checked the per-host-numa-node 2MB huge page availability
> > on your host?
> >   If it's uneven then that might explain what you're seeing.
> >
> >
> > These are the observations/questions I have:
> >
> > 1) On the host, I was seeing 32768 huge pages, of 2MB size. When I
> > created VMs (Cirros) using small flavor, each VM was getting created on
> > NUMA nodeid 0. When it hit half of the available pages, I could no
> > longer create any VMs (QEMU saying no space). I'd like to understand why
> > the assignment was always going two nodeid 0, and to confirm that the
> > huge pages are divided among the number of NUMA nodes available.
> >
> > 2) I changed mem_page_size from 1024 to 2048 in the flavor, and then
> > when VMs were created, they were being evenly assigned to the two NUMA
> > nodes. Each using 1024 huge pages. At this point I could create more
> > than half, but when there were 1945 pages left, it failed to create a
> > VM. Did it fail because the mem_page_size was 2048 and the available
> > pages were 1945, even though we were only requesting 1024 pages?
> >
> > 3) Related to #2, is there a relationship between mem_page_size, the
> > allocation of VMs to NUMA nodes, and the flavor size? IOW, if I use the
> > medium flavor (4GB), will I need a larger mem_page_size? (I'll play with
> > this variation, as soon as I can). Gets back to understanding how the
> > scheduling determines how to assign the VMs.
> >
> > 4) When the VM create failed due to QEMU failing allocation, the VM went
> > to error state. I deleted the VM, but the neutron port was still there,
> > and there were no log messages indicating that a request was made to
> > delete the port. Is this expected (that the user would have to manually
> > clean up the port)?
>
> When you hit this case, can you check if instance.host is set in the
> database before deleting the instance? I'm guessing what's happening is
> the instance didn't get assigned a host since it eventually ended up
> with NoValidHost, so when you go to delete it doesn't have a compute to
> send it to for delete, so it deletes from the compute API, and we don't
> have the host binding details to delete the port.
>
> Although, when the spawn failed in the compute to begin with we should
> have deallocated any networking that was created before kicking back to
> the scheduler - unless we don't go back to the scheduler if the instance
> is set to ERROR state.
>
> A bug report with stacktrace of the failure scenario when the instance
> goes to error state bug n-cpu logs would probably help.
>
> >
> > 5) A coworker had hit the problem mentioned in #1, with exhaustion at
> > the halfway point. If she delete's a VM, and then changes the flavor to
> > change the mem_page_size to 2048, should Nova start assigning all new
> > VMs to the other NUMA node, until the pool of huge pages is down to
> > where the huge pages are for NUMA node 0, or will it alternate between
> > the available NUMA nodes (and run out when node 0's pool is exhausted)?
> >
> > Thanks in advance!
> >
> > PCM
> >
> >
> >
> >
> > Chris
> >
> >
>  __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > <
> http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> --
>
> Thanks,
>
> 

Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Sean Dague
On 06/10/2016 08:41 AM, Jeremy Stanley wrote:
> On 2016-06-10 11:49:12 +0100 (+0100), Miles Gould wrote:
>> On 09/06/16 23:21, Jay Faulkner wrote:
>>> There was some discussion about whether or not the Ironic grenade job
>>> should be in the check pipeline (even as -nv) for grenade,
>>
>> Not having this would mean that changes to grenade could silently break
>> Ironic's CI, right? That sounds really bad.
> 
> That's like saying it's really bad that changes to devstack could
> silently break devstack-based jobs for random projects, and so they
> should be tested against every one of those jobs. At some point you
> have to draw a line between running a reasonably representative
> sample and running so many jobs that you'll never be able to merge
> another change again (because even very small nondeterministic
> failure rates compound to make that impossible at a certain scale).

Nothing should be voting in check in grenade that requires a plugin.

I'm fine with a few things in check nv if they are doing something out
of the ordinary that we think needs to be kept on top of. I also expect
that ironic folks are going to watch for those failures, and say, with
-1/+1 CR, when they are legit and when it was off the rails. A non
voting job that doesn't have domain experts validating the content
regularly with CR means it gets ignored if it fails a bunch.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [openstack-ansible] Mid-cycle date selection (need input!)

2016-06-10 Thread Major Hayden
On 06/10/2016 03:16 AM, Jesse Pretorius wrote:
> Thanks Major. I have no conflicts for any of the dates. By option 2 I'm 
> guessing you mean either 22-24 August (Mon-Wed) or 24-26 (Wed-Fri) rather 
> than the entire week?

Correct.  For the August 22-26 dates, we could choose anything within that 
range.  I'm not sure if hugging a weekend or sitting in the middle of the week 
is best for us.  I'd imagine that folks outside the US might appreciate a 
weekend to recover from time zone changes before or after.

--
Major Hayden

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Jeremy Stanley
On 2016-06-10 11:49:12 +0100 (+0100), Miles Gould wrote:
> On 09/06/16 23:21, Jay Faulkner wrote:
> >There was some discussion about whether or not the Ironic grenade job
> >should be in the check pipeline (even as -nv) for grenade,
> 
> Not having this would mean that changes to grenade could silently break
> Ironic's CI, right? That sounds really bad.

That's like saying it's really bad that changes to devstack could
silently break devstack-based jobs for random projects, and so they
should be tested against every one of those jobs. At some point you
have to draw a line between running a reasonably representative
sample and running so many jobs that you'll never be able to merge
another change again (because even very small nondeterministic
failure rates compound to make that impossible at a certain scale).
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][glance][qa] - Nova glance v2 work complete

2016-06-10 Thread Kairat Kushaev
\0/
That's awesome.
Big thanks to mfedosin and sudipto for driving this work.

Best regards,
Kairat Kushaev

On Fri, Jun 10, 2016 at 2:52 PM, Monty Taylor  wrote:

> On 06/10/2016 01:19 PM, Sean Dague wrote:
> > On 06/07/2016 04:55 PM, Matt Riedemann wrote:
> >> I tested the glance v2 stack (glance v1 disabled) using a devstack
> >> change here:
> >>
> >> https://review.openstack.org/#/c/325322/
> >>
> >> Now that the changes are merged up through the base nova image proxy and
> >> the libvirt driver, and we just have hyper-v/xen driver changes for that
> >> series, we should look at gating on this configuration.
> >>
> >> I was originally thinking about adding a new job for this, but it's
> >> probably better if we just change one of the existing integrated gate
> >> jobs, like gate-tempest-dsvm-full or gate-tempest-dsvm-neutron-full.
> >>
> >> Does anyone have an issue with that? Glance v1 is deprecated and the
> >> configuration option added to nova (use_glance_v1) defaults to True for
> >> compat but is deprecated, and the Nova team plans to drop it's v1 proxy
> >> code in Ocata. So it seems like changing config to use v2 in the gate
> >> jobs should be a non-issue. We'd want to keep at least one integrated
> >> gate job using glance v1 to make sure we don't regress anything there in
> >> Newton.
> >
> > use_glance_v1=False has now been merged as the default, so all jobs are
> > now using glance v2 for the Nova <=> Glance communication -
> > https://review.openstack.org/#/c/321551/
> >
> > Thanks to Mike and Sudipta for pushing this to completion.
>
> Congrats everybody!!!
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Jim Rollenhagen
On Thu, Jun 09, 2016 at 03:21:05PM -0700, Jay Faulkner wrote:
> A quick update:
> 
> The devstack-gate patch is currently merging.
> 
> There was some discussion about whether or not the Ironic grenade job should
> be in the check pipeline (even as -nv) for grenade, so I split that patch
> into two pieces so the less controversial part (adding the grenade-nv job to
> Ironic's check pipeline) could merge more easily.
> 
> https://review.openstack.org/#/c/319336/ - project-config
> Make grenade-dsvm-ironic non voting (in the check queue for Ironic only)
> 
> https://review.openstack.org/#/c/327985/ - project-config
> Make grenade-dsvm-ironic non voting (in the check queue for grenade)
> 
> Getting working upgrade testing will be a huge milestone for Ironic. Thanks
> to those who have already helped us make progress and those who will help us
> land these and see it at work.

The first of these is merged, which gets us most of what we needed.
Thanks everyone for your hard work on this!

/me rechecks some things waiting for this testing :D

// jim

> 
> Thanks in advance,
> Jay Faulkner
> OSIC
> 
> On 6/9/16 8:28 AM, Jim Rollenhagen wrote:
> >Hi friends,
> >
> >We're two patches away from having grenade passing in our check queue!
> >This is a huge step forward for us, many thanks go to the numerous folks
> >that have worked on or helped somehow with this.
> >
> >I'd love to push this across the line today as it's less than 10 lines
> >of changes between the two, and we have a bunch of work nearly done that
> >we'd like upgrade testing running against before merging.
> >
> >So we need infra cores' help here.
> >
> >https://review.openstack.org/#/c/316662/ - devstack-gate
> >Allow to pass OS_TEST_TIMEOUT for grenade job
> >1 line addition with an sdague +2.
> >
> >https://review.openstack.org/#/c/319336/ - project-config
> >Make grenade-dsvm-ironic non voting (in the check queue)
> >+7,-1 with an AJaeger +2.
> >
> >Thanks in advance. :)
> >
> >// jim
> >
> >__
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][glance][qa] - Nova glance v2 work complete

2016-06-10 Thread Monty Taylor
On 06/10/2016 01:19 PM, Sean Dague wrote:
> On 06/07/2016 04:55 PM, Matt Riedemann wrote:
>> I tested the glance v2 stack (glance v1 disabled) using a devstack
>> change here:
>>
>> https://review.openstack.org/#/c/325322/
>>
>> Now that the changes are merged up through the base nova image proxy and
>> the libvirt driver, and we just have hyper-v/xen driver changes for that
>> series, we should look at gating on this configuration.
>>
>> I was originally thinking about adding a new job for this, but it's
>> probably better if we just change one of the existing integrated gate
>> jobs, like gate-tempest-dsvm-full or gate-tempest-dsvm-neutron-full.
>>
>> Does anyone have an issue with that? Glance v1 is deprecated and the
>> configuration option added to nova (use_glance_v1) defaults to True for
>> compat but is deprecated, and the Nova team plans to drop it's v1 proxy
>> code in Ocata. So it seems like changing config to use v2 in the gate
>> jobs should be a non-issue. We'd want to keep at least one integrated
>> gate job using glance v1 to make sure we don't regress anything there in
>> Newton.
> 
> use_glance_v1=False has now been merged as the default, so all jobs are
> now using glance v2 for the Nova <=> Glance communication -
> https://review.openstack.org/#/c/321551/
> 
> Thanks to Mike and Sudipta for pushing this to completion.

Congrats everybody!!!


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][glance][qa] - Nova glance v2 work complete

2016-06-10 Thread Sean Dague
On 06/07/2016 04:55 PM, Matt Riedemann wrote:
> I tested the glance v2 stack (glance v1 disabled) using a devstack
> change here:
> 
> https://review.openstack.org/#/c/325322/
> 
> Now that the changes are merged up through the base nova image proxy and
> the libvirt driver, and we just have hyper-v/xen driver changes for that
> series, we should look at gating on this configuration.
> 
> I was originally thinking about adding a new job for this, but it's
> probably better if we just change one of the existing integrated gate
> jobs, like gate-tempest-dsvm-full or gate-tempest-dsvm-neutron-full.
> 
> Does anyone have an issue with that? Glance v1 is deprecated and the
> configuration option added to nova (use_glance_v1) defaults to True for
> compat but is deprecated, and the Nova team plans to drop it's v1 proxy
> code in Ocata. So it seems like changing config to use v2 in the gate
> jobs should be a non-issue. We'd want to keep at least one integrated
> gate job using glance v1 to make sure we don't regress anything there in
> Newton.

use_glance_v1=False has now been merged as the default, so all jobs are
now using glance v2 for the Nova <=> Glance communication -
https://review.openstack.org/#/c/321551/

Thanks to Mike and Sudipta for pushing this to completion.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic][infra][qa] Ironic grenade work nearly complete

2016-06-10 Thread Miles Gould

On 09/06/16 23:21, Jay Faulkner wrote:

There was some discussion about whether or not the Ironic grenade job
should be in the check pipeline (even as -nv) for grenade,


Not having this would mean that changes to grenade could silently break 
Ironic's CI, right? That sounds really bad.


Miles

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [charms] Re-licensing OpenStack charms under Apache 2.0

2016-06-10 Thread Neil Jerram
Me too.

On Fri, Jun 10, 2016 at 10:32 AM Cory Benfield  wrote:

>
> > On 8 Jun 2016, at 11:20, James Page  wrote:
> >
> > The majority of contributors are from Canonical (from whom I have
> permission to make this switch) with a further 18 contributors from outside
> of Canonical who I will be directly contacting for approval in gerrit as
> reviews are raised for each repository.
>
> Hey James,
>
> I’m happy for you to relicense my contributions as Apache 2.0.
>
> Cory
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron][networking-ovn] OVN vs. OpenDayLight

2016-06-10 Thread Oleg Bondarev
Hi,

[1] is a doc comparing OVN and ODL for Neutron. I wrote it in March so some
info might be stale.
Hope this can be useful, comments are welcome!

[1]
https://docs.google.com/document/d/13K4Xpdu1IbRJDj5JTepDI4nwUkCDCu7IOnYwHkZtcMA/edit?usp=sharing

Thanks,
Oleg

On Fri, Jun 10, 2016 at 12:28 AM, Kyle Mestery  wrote:

> On Thu, Jun 9, 2016 at 4:19 PM, Assaf Muller  wrote:
> > On Thu, Jun 9, 2016 at 5:06 PM, Kyle Mestery 
> wrote:
> >> On Thu, Jun 9, 2016 at 2:11 PM, Assaf Muller  wrote:
> >>> On Thu, Jun 9, 2016 at 1:48 PM, Ben Pfaff  wrote:
>  On Thu, Jun 09, 2016 at 10:28:31AM -0700, rezroo wrote:
> > I'm trying to reconcile differences and similarities between OVN and
> > OpenDayLight in my head. Can someone help me compare these two
> technologies
> > and explain if they solve the same problem, or if there are
> fundamental
> > differences between them?
> 
>  OVN implements network virtualization for clouds of VMs or containers
> or
>  a mix.  Open Daylight is a platform for managing networks that can do
>  anything you want.
> >>>
> >>> That is true, but when considering a Neutron backend for OpenStack
> >>> deployments, people choose a subset of OpenDaylight projects and the
> >>> end result is a solution that is comparable in scope and feature set.
> >>> There are objective differences in where the projects are in their
> >>> lifetime, the HA architecture, the project's consistency model between
> >>> the neutron-server process and the backend, the development velocity,
> >>> the community size and the release model.
> >>>
> >> Fundamentally, the main difference is that OVN does one thing: It does
> >> network virtualization. OpenDaylight _MAY_ do network virtualization,
> >> among other things, and it likely does network virtualization in many
> >> different ways. Like Ben said:
> >>
> >> "Open Daylight is a platform for managing networks that can do
> >> anything you want."
> >
> > I agree, but I don't think that was what was asked or makes for an
> > interesting discussion. I think the obvious comparison is OVN to
> > ML2/ODL using the ovsdb ODL project.
> >
> OK, I'll bite. :)
>
> Fundamentally, a project's focus is absolutely important, especially
> when a comparison is asked. When you ask the question: "How can OVN or
> ODL solve being a backend layer for Neutron?", for example, the answer
> with OVN is simple: You do it this way, and it works. For ODL, the
> question is much more nuanced, as it depends on *what* components in
> ODL you are using.
>
> Also, yes, the comparison between "ML2+python agents" vs. "ML2+OVN" is
> much more relevant IMHO.
>
> Thanks!
> Kyle
>
> >>
> >> Thanks,
> >> Kyle
> >>
> 
> 
> __
>  OpenStack Development Mailing List (not for usage questions)
>  Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>
> __
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [magnum-ui][magnum] Proposed Core addition, and removal notice

2016-06-10 Thread Shuu Mutou
Hi team,

I propose the following changes to the magnum-ui core group.

+ Thai Tran
  http://stackalytics.com/report/contribution/magnum-ui/90
  I'm so happy to propose Thai as a core reviewer.
  His reviews have been extremely valuable for us. 
  And he is active Horizon core member.
  I believe his help will lead us to the correct future.

- David Lyle
  
http://stackalytics.com/?metric=marks_type=openstack=all=magnum-ui_id=david-lyle
  No activities for Magnum-UI since Mitaka cycle.

- Harsh Shah
  http://stackalytics.com/report/users/hshah
  No activities for OpenStack in this year.

- Ritesh
  http://stackalytics.com/report/users/rsritesh
  No activities for OpenStack in this year.

Please respond with your +1 votes to approve this change or -1 votes to oppose.

Thanks,
Shu


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [charms] Re-licensing OpenStack charms under Apache 2.0

2016-06-10 Thread Cory Benfield

> On 8 Jun 2016, at 11:20, James Page  wrote:
> 
> The majority of contributors are from Canonical (from whom I have permission 
> to make this switch) with a further 18 contributors from outside of Canonical 
> who I will be directly contacting for approval in gerrit as reviews are 
> raised for each repository.

Hey James,

I’m happy for you to relicense my contributions as Apache 2.0.

Cory



signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] theoretical race between live migration and resource audit?

2016-06-10 Thread lương hữu tuấn
Hi,

Yes, it is actually a race and we have already faced a negative effect when
using evacuation. Some information of cpu pinning is lost. Imagine that, in
some cases, we do some re-scheduling actions (evacuate, live-migration,
etc.) then immediately do the next actions (delete, resize, etc.) before
the resource_tracker updates in the next period. In this case, it fails.
Actually, it has some negative side in writing tests based on the scenario
described above.

Br,

Tutj

On Fri, Jun 10, 2016 at 10:36 AM, Matthew Booth  wrote:

> Yes, this is a race.
>
> However, it's my understanding that this is 'ok'. The resource tracker
> doesn't claim to be 100% accurate at all times, right? Otherwise why would
> it update itself in a period task in the first place. It's my understanding
> that the resource tracker is basically a best effort cache, and that
> scheduling decisions can still fail at the host. The resource tracker will
> fix itself next time it runs via its periodic task.
>
> Matt (not a scheduler person)
>
> On Thu, Jun 9, 2016 at 10:41 PM, Chris Friesen <
> chris.frie...@windriver.com> wrote:
>
>> Hi,
>>
>> I'm wondering if we might have a race between live migration and the
>> resource audit.  I've included a few people on the receiver list that have
>> worked directly with this code in the past.
>>
>> In _update_available_resource() we have code that looks like this:
>>
>> instances = objects.InstanceList.get_by_host_and_node()
>> self._update_usage_from_instances()
>> migrations = objects.MigrationList.get_in_progress_by_host_and_node()
>> self._update_usage_from_migrations()
>>
>>
>> In post_live_migration_at_destination() we do this (updating the host and
>> node as well as the task state):
>> instance.host = self.host
>> instance.task_state = None
>> instance.node = node_name
>> instance.save(expected_task_state=task_states.MIGRATING)
>>
>>
>> And in _post_live_migration() we update the migration status to
>> "completed":
>> if migrate_data and migrate_data.get('migration'):
>> migrate_data['migration'].status = 'completed'
>> migrate_data['migration'].save()
>>
>>
>> Both of the latter routines are not serialized by the
>> COMPUTE_RESOURCE_SEMAPHORE, so they can race relative to the code in
>> _update_available_resource().
>>
>>
>> I'm wondering if we can have a situation like this:
>>
>> 1) migration in progress
>> 2) We start running _update_available_resource() on destination, and we
>> call instances = objects.InstanceList.get_by_host_and_node().  This will
>> not return the migration, because it is not yet on the destination host.
>> 3) The migration completes and we call
>> post_live_migration_at_destination(), which sets the host/node/task_state
>> on the instance.
>> 4) In _update_available_resource() on destination, we call migrations =
>> objects.MigrationList.get_in_progress_by_host_and_node().  This will return
>> the migration for the instance in question, but when we run
>> self._update_usage_from_migrations() the uuid will not be in "instances"
>> and so we will use the instance from the newly-queried migration.  We will
>> then ignore the instance because it is not in a "migrating" state.
>>
>> Am I imagining things, or is there a race here?  If so, the negative
>> effects would be that the resources of the migrating instance would be
>> "lost", allowing a newly-scheduled instance to claim the same resources
>> (PCI devices, pinned CPUs, etc.)
>>
>> Chris
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Matthew Booth
> Red Hat Engineering, Virtualisation Team
>
> Phone: +442070094448 (UK)
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [swift][keystone] Using JSON as future ACL format

2016-06-10 Thread Coles, Alistair
Thai,

(repeating some of what we have discussed in private for others' benefit)

The Swift docs state "Be sure to use Keystone UUIDs rather than names in 
container ACLs" [1]. The guidance is re-iterated here [2] "...names must no 
longer be used in cross-tenant ACLs...". They then go on to explain that for 
backwards compatibility (i.e. to not break ACLS that have already been 
persisted in Swift) names in ACLs are supported in the default domain only.

The intent was not to encourage the continued use of names in the default 
domain (or while using keystone v2), nor to suggest that was "safe", but to 
recognize that names had previously been used in contexts where names were 
unique and the ':' separator was safe. In fact, Swift provides an option to 
disallow name matching in *all* contexts when no such backwards compatibility 
is required.

At the time we made those changes (c. Atlanta summit) the input I had from 
Keystone devs was that names were not globally unique (with keystone v3), were 
mutable and should not be persisted. Hence the swift docs guidance. We actually 
considered preventing any new ACL being set with names, but to do so requires 
distinguishing a name string from a UUID string, which we didn't find a 
solution for.

So in response to your argument "If we are allowing V2 to store names [{ 
project, name }], I do not see why we should not allow the same for V3 [{ 
domain, project, name }]" : yes, we are allowing names in ACLs in some 
circumstances, but only for backwards compatibility, and we are not encouraging 
it. Having gone through the pain of dealing with names in existing persisted 
ACLs, I am reluctant to encourage their continued/renewed use.

Are their examples of any other projects requiring names rather than UUIDs in 
ACLs, or for other purposes, that we can learn from?

The idea discussed here [3] (not implemented) was that names could be supported 
in a JSON ACL format but should be resolved to UUIDs before persisting in 
Swift. That way a user's name can change but since their request token is 
resolved to UUID then any persisted ACL would still match. As has already been 
mentioned in another reply on this thread, Swift has a JSON ACL format for 
"v1"/TempAuth account level ACLs [4] that could perhaps be implemented for 
keystoneauth and then extended to containers.

Alistair

[1] 
http://docs.openstack.org/developer/swift/overview_auth.html#access-control-using-keystoneauth
[2] http://docs.openstack.org/developer/swift/middleware.html#keystoneauth
[3] https://wiki.openstack.org/wiki/Swift/ContainerACLWithKeystoneV3
[4] http://docs.openstack.org/developer/swift/overview_auth.html#tempauth



From: Thai Q Tran [mailto:tqt...@us.ibm.com]
Sent: 06 June 2016 21:06
To: OpenStack Development Mailing List (not for usage questions) 

Subject: [openstack-dev] [swift][keystone] Using JSON as future ACL format


Hello all,

Hope everyone had a good weekend, and hope this email does not ruin your next.
We had a small internal discussion at IBM and here are some of the findings 
that I will present to the wider community.

1. The ":" separator that swift currently uses is not entirely safe since LDAP 
can be configured to allow special characters in user IDs. It essentially means 
no special characters are safe to use as separators. I am not sure how 
practical this is, but its something to consider.

2. Since names are not guaranteed to be immutable, we should store everything 
via IDs. Currently, for backward compatibility reasons, Swift continues to 
support names for for V2. Keep in mind that V2 does not guarantee that names 
are immutable either. Given this fact and what we know from #1, we can say that 
names are mutable for both V2 and V3, and that any separators we use are 
fallible. In other words, using a separator for names or ids will not work 100% 
of the time.

3. Keystone recently enabled URL safe naming of project and domains for their 
hierarchal work. As a by product of that, if the option is enabled, Swift can 
essentially use the reserved characters as separators. The list of reserved 
characters are listed below. The only question remaining, how does Keystone 
inform Swift that this option is enabled? or Swift can add an separator option 
that is a subset of the characters below and leave it to the deployer to 
configure it.

";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" |"$" | ","

https://github.com/openstack/keystone/commit/60b52c1248ddd5e682838d9e8ba853789940c284
http://www.ietf.org/rfc/rfc2396.txt

3. As mentioned in the KeystoneAuthACL write up in Atlanta, JSON format is one 
of the option going forward. The article goes on to mention that we should 
store only user IDs (avoiding the mutable names issue). It outlined a process 
and reverse-process that would allow names to be use but mentioned an overhead 
cost to Keystone. I personally think is the right approach going forward since 
it negate the use of a separator 

Re: [openstack-dev] [release][reno][infra] merging tags between branches is confusing our release notes

2016-06-10 Thread Thierry Carrez

Doug Hellmann wrote:

Excerpts from John Dickinson's message of 2016-06-08 11:30:03 -0700:

Isn't the reason that the branch is merged back in because otherwise per can't 
generate a valid version number?


I don't think it's related to versions being "valid," but to making
things feel less confusing to human consumers at the expense of
(what I think is) giving a misleading picture of the version history.

We started merging the tags between branches because the version
that ends up in the tarball generated from patches merging into
master is lower than the version in the tarball generated from the
stable branch, after the stable branch has received the "final" tag
for the release (master still has a 0rc1 tag for the same version
that is final on the branch). This is confusing to humans, but if
those humans have not set up automated systems to install or otherwise
consume artifacts for the same project from multiple branches it
should not confuse any computers.


IIRC we started merging tags back so that the humans would not be 
surprised that "git tags" would return something like:


...
12.0.0.0b3
12.0.0.0rc1
13.0.0.0b1
...

i.e. the release would never be there. This was especially confusing 
back when we used an hybrid release model for Swift, where intermediary 
releases would be directly tagged but the "final" one would use a 
release candidates on a release branch, resulting in something weird 
like this when looking at master:


...
1.5.0
1.5.1
1.5.2rc1
1.6.0
...

We don't do this hybrid anymore for intermediary-released projects, 
everything is tagged directly. So you might still be missing stable 
release tags when running git tags on master, but while that may still 
be slightly confusing to humans, it's not as bad as the weird output above.


--
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] NUMA, huge pages, and scheduling

2016-06-10 Thread Daniel P. Berrange
On Thu, Jun 09, 2016 at 12:35:06PM -0600, Chris Friesen wrote:
> On 06/09/2016 05:15 AM, Paul Michali wrote:
> > 1) On the host, I was seeing 32768 huge pages, of 2MB size.
> 
> Please check the number of huge pages _per host numa node_.
> 
> > 2) I changed mem_page_size from 1024 to 2048 in the flavor, and then when 
> > VMs
> > were created, they were being evenly assigned to the two NUMA nodes. Each 
> > using
> > 1024 huge pages. At this point I could create more than half, but when there
> > were 1945 pages left, it failed to create a VM. Did it fail because the
> > mem_page_size was 2048 and the available pages were 1945, even though we 
> > were
> > only requesting 1024 pages?
> 
> I do not think that "1024" is a valid page size (at least for x86).

Correct, 4k, 2M and 1GB are valid page sizes.

> Valid mem_page_size values are determined by the host CPU.  You do not need
> a larger page size for flavors with larger memory sizes.

Though note that page sizes should be a multiple of favour mem size
unless you want to waste memory. eg if you have a flavour with 750MB
RAM, then you probably don't want to use 1GB pages as it waste 250MB

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Neutron] Random IP address allocations

2016-06-10 Thread Gary Kotton
Hi,
The patch https://review.openstack.org/#/c/292207/ has broken decomposed 
plugins. I am not sure if we can classify this as a API change - basically the 
IP address allocation model has changed. So someone prior to the patch could 
create a network and expect addresses A, B and C to be allocated in that order. 
Now random addresses will be allocated.
I think that this requires some discussion and we should consider reverting the 
patch. Maybe I am missing something here but this may break people who are 
working according the existing outputs of Neutron according to existing 
behavior (which may have been wrong from the start).
Any thoughts?
Thanks
Gary
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] theoretical race between live migration and resource audit?

2016-06-10 Thread Matthew Booth
Yes, this is a race.

However, it's my understanding that this is 'ok'. The resource tracker
doesn't claim to be 100% accurate at all times, right? Otherwise why would
it update itself in a period task in the first place. It's my understanding
that the resource tracker is basically a best effort cache, and that
scheduling decisions can still fail at the host. The resource tracker will
fix itself next time it runs via its periodic task.

Matt (not a scheduler person)

On Thu, Jun 9, 2016 at 10:41 PM, Chris Friesen 
wrote:

> Hi,
>
> I'm wondering if we might have a race between live migration and the
> resource audit.  I've included a few people on the receiver list that have
> worked directly with this code in the past.
>
> In _update_available_resource() we have code that looks like this:
>
> instances = objects.InstanceList.get_by_host_and_node()
> self._update_usage_from_instances()
> migrations = objects.MigrationList.get_in_progress_by_host_and_node()
> self._update_usage_from_migrations()
>
>
> In post_live_migration_at_destination() we do this (updating the host and
> node as well as the task state):
> instance.host = self.host
> instance.task_state = None
> instance.node = node_name
> instance.save(expected_task_state=task_states.MIGRATING)
>
>
> And in _post_live_migration() we update the migration status to
> "completed":
> if migrate_data and migrate_data.get('migration'):
> migrate_data['migration'].status = 'completed'
> migrate_data['migration'].save()
>
>
> Both of the latter routines are not serialized by the
> COMPUTE_RESOURCE_SEMAPHORE, so they can race relative to the code in
> _update_available_resource().
>
>
> I'm wondering if we can have a situation like this:
>
> 1) migration in progress
> 2) We start running _update_available_resource() on destination, and we
> call instances = objects.InstanceList.get_by_host_and_node().  This will
> not return the migration, because it is not yet on the destination host.
> 3) The migration completes and we call
> post_live_migration_at_destination(), which sets the host/node/task_state
> on the instance.
> 4) In _update_available_resource() on destination, we call migrations =
> objects.MigrationList.get_in_progress_by_host_and_node().  This will return
> the migration for the instance in question, but when we run
> self._update_usage_from_migrations() the uuid will not be in "instances"
> and so we will use the instance from the newly-queried migration.  We will
> then ignore the instance because it is not in a "migrating" state.
>
> Am I imagining things, or is there a race here?  If so, the negative
> effects would be that the resources of the migrating instance would be
> "lost", allowing a newly-scheduled instance to claim the same resources
> (PCI devices, pinned CPUs, etc.)
>
> Chris
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Matthew Booth
Red Hat Engineering, Virtualisation Team

Phone: +442070094448 (UK)
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [openstack-ansible] Mid-cycle date selection (need input!)

2016-06-10 Thread Jesse Pretorius
On 9 June 2016 at 19:51, Major Hayden  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Hey folks,
>
> I've been able to secure a few dates at Rackspace's headquarters in San
> Antonio, Texas:
>
>   1) August 10-12
>   2) August 22-26
>   3) August 29 - September 2
>
> During the meeting earlier today, #3 was determined to cause a lot of
> conflicts for people.  #1 seems to be the most preferred.  I have emails
> out to ask about deals on local hotels and I'm waiting to hear back on
> those.
>
> The room should seat about 20-25 people and we would have at least one
> projector.
>
> Please reply with your thoughts and a date preference!  Once we get that
> sorted out, we can fire up an etherpad for everyone to sign up for a spot.
>

Thanks Major. I have no conflicts for any of the dates. By option 2 I'm
guessing you mean either 22-24 August (Mon-Wed) or 24-26 (Wed-Fri) rather
than the entire week?

I would prefer an option in option 2's range, but option 1 or 3 are also
fine.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev