Re: [openstack-dev] [qa] Use test scenarios for interface type and auth version

[Robert Collins]

On 4 January 2014 04:14, Frittoli, Andrea (Cloud Services)
 wrote:
> Hi,
>
>
>
> I’m working on multi-keystone-api-version-tests which aims to provide the
> ability to run tests using a configurable list of keystone API versions for
> authentication.
>
>
>
> If two versions of the API are configured, each test would be run using the
> two versions.
>
> In the gate, where we’re short of time, we could test only the latest stable
> version, but in other tests will have the ability to use more than one
> version.
>
>
>
> The approach I’m considering is based on testscenarios.
>
> Thinking about how to implement this I recognized that a solution that works
> for auth version could be used also to run json / xml, removing the need to
> have two classes for API test.

+1.

> The idea I have is to change Base[Service]Test classes and move the client
> selection logic to BaseTestCase.  Something like this:
>
>
>
> def __getattr__(self, attr):
>
> if attr in
> manager.list_of_available_clients:
>
> manager.get_client(attr,
> self.auth_version, self.interface)
>
>   else:
>
>  raise AttributeError


Ugh, don't override __getattr__ (or __getattribute__) for something
like this. ETOOMUCHMAGIC.

def self.get_client(self, attr):
 

Will be much more obvious.

Cheers,
Rob
-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Robert Collins]

On 4 January 2014 08:44, Doug Hellmann  wrote:
>
>
>

> I understand that moving taskflow into oslo would avoid the policy decision
> we have in place to not do symmetric gating on unreleased versions of things
> not "owned" by the OpenStack project. However, I don't know if we want to be
> testing against the git head of libraries no matter where they live. As
> fungi pointed out on IRC, gating against pre-release versions of libraries
> may allow us to reach a state where the software works when installed from
> git, but not from the released packages.

We've certainly hit that situation with nova and neutron where there
were undetected, undocumented issues between trunk server and
last-release client. [TripleO deploys with requirements.txt
requirements, not 'trunk', of each project...] because when we deploy
a released server, at the time we set this up, there was no way to
identify the matching trunk revisions. We could change this now, but
it's been useful to identify unreleased client code being used as a
hard dep in servers - a bad situation.

> It seems safer to gate changes to libraries against the apps' trunk (to
> avoid making backwards-incompatible changes), and then gate changes to the
> apps against the released libraries (to ensure they work with something
> available to be packaged by the distros). There are lots and lots of version
> numbers available to us, so I see no problem with releasing new versions of
> libraries frequently.

+1.

I do see the issue Sean is pointing at, which is that we have to fix
the libraries first and then the things that use them. OTOH thats
normal in the software world, I don't see anything unique about it.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Joshua Harlow]

So another idea that was talked about on IRC.

Taskflow exposes entrypoints for these storage backends (like your storage
callback/interface idea).

It currently provides 3 such 'default' backends [sqlalchemy, file/dir
based, in-memory <--> mainly for testing].

A 4th one is in progress for icehouse (zookeeper based).

These backends are used to store intermediary 'flow/task' state (allowing
the taskflow engine to resume if an app crashes/stopped while doing stuff).

A couple ideas about this, since its already pluggable.

Split the sqlalchemy backend -> 'taskflow-sa' repo/new package. For those
projects that want to use this backend, they include it (still means
'taskflow-sa' package has a dependency on sqlalchemy of some version).
Another idea is to move the sqlalchemy dependency currently in taskflow
requirements.txt -> taskflow test-requirements.txt and for those projects
that want to use the sqlalchemy backend, they include the sqlalchemy
version themselves in there requirements.txt (taskflow keeps it in
test-requirements.txt so that it can run its unit/integration tests,
making sure the backend still works).

I'm not really sure which is the best, none seem super-great, but
sometimes u just work with what u got :-P

On 1/3/14, 3:32 PM, "Sean Dague"  wrote:

>On 01/03/2014 06:14 PM, Joshua Harlow wrote:
>> Since the library model is what most everyone else uses outside of
>> openstack (I assume?) what can we do to get there so that this model
>>works
>> as well?
>>
>> Expanding dependencies recursively seems like it could help? This could
>> then detect transitive dependency issues (and doesn't seem so hard to
>>do).
>
>We actually talked about having pip be able to help us here with a
>--requirements-override piece of function. dstufft seemed positive on
>the concept.
>
>> I like the idea of 'gate on all the things' (that I've heard come up
>> before) but I don't know if its possible? If we hooked into the pypi
>> upload 'stream' it would seem like we could automatically trigger
>> openstack builds when a known dependency (or dependency of a
>> dependency...) is uploaded (maybe?). That would be pretty neat.
> >
>> In general it really seems like having more libraries, not less is ideal
>> (making us solve this issue whether we want to or not really). As
>> libraries can then be used outside of openstack (taskflow is already
>>being
>> used elsewhere also), libraries create well-defined apis and boundaries
>> between programs (...). I know they also create this dependency hell
>> problem (anvil has been hitting these same issues for a while to). But I
>> think we can figure out a solution that fits both worlds, the world of
>> things we can gate on and the world of things we can't (3rd party
>> libraries that aren't under openstack control). Taskflow is in-between
>> those worlds, so it allows us to explore how to make both of those
>>worlds
>> work.
>
>In general I agree however, if you get between OpenStack and SQLA
>you've now touched the 3rd rail. Because we have deep experience about
>how bad the compatibility between versions can be, and we can't be
>beholden to another project about our SQLA upgrade timeframe.
>
>So I think that generalities aside, if are a library, and use SQLA, we
>probably need to really think about putting it in the integrated gate.
>
>Because otherwise what we are saying is taskflow is completely dictating
>the SQLA version in the Icehouse release of OpenStack. And that's the
>wrong place for that decision to be.
>
>If taskflow worked with a storage callback mechanism, and got a storage
>interface from the program that was calling it, then things would be
>much different. But because it's going straight to the database and
>managing tables directly, through a known unstable library, that
>OpenStack itself needs some control over, it's definitely a different
>case.
>
>   -Sean
>
>-- 
>Sean Dague
>Samsung Research America
>s...@dague.net / sean.da...@samsung.com
>http://dague.net
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [trove] mid cycle meetup

[Michael Basnight]

Howdy,

In the spirit of getting together to get stuff done, Trove is having a mid
cycle meetup in Austin, TX. We are having it Wednesday, February 19, 2014
to Friday, February 21, 2014, in downtown Austin at the Capital Factory,
which is a hip startup incubator. See [1] for more information.

The agenda is still somewhat up in the air, but we will be covering
clustering, trove tempest testing, and heat integration at present. To me,
this will be focused on part design + part implementation of specific
features, so if you have an agenda item please bring it up! Ill be
constructing a loosy goosy agenda in the coming weeks.

[1] https://wiki.openstack.org/wiki/Trove/IcehouseCycleMeetup
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

On 01/03/2014 06:14 PM, Joshua Harlow wrote:

Since the library model is what most everyone else uses outside of
openstack (I assume?) what can we do to get there so that this model works
as well?

Expanding dependencies recursively seems like it could help? This could
then detect transitive dependency issues (and doesn't seem so hard to do).


We actually talked about having pip be able to help us here with a 
--requirements-override piece of function. dstufft seemed positive on 
the concept.



I like the idea of 'gate on all the things' (that I've heard come up
before) but I don't know if its possible? If we hooked into the pypi
upload 'stream' it would seem like we could automatically trigger
openstack builds when a known dependency (or dependency of a
dependency...) is uploaded (maybe?). That would be pretty neat.

>

In general it really seems like having more libraries, not less is ideal
(making us solve this issue whether we want to or not really). As
libraries can then be used outside of openstack (taskflow is already being
used elsewhere also), libraries create well-defined apis and boundaries
between programs (...). I know they also create this dependency hell
problem (anvil has been hitting these same issues for a while to). But I
think we can figure out a solution that fits both worlds, the world of
things we can gate on and the world of things we can't (3rd party
libraries that aren't under openstack control). Taskflow is in-between
those worlds, so it allows us to explore how to make both of those worlds
work.


In general I agree however, if you get between OpenStack and SQLA 
you've now touched the 3rd rail. Because we have deep experience about 
how bad the compatibility between versions can be, and we can't be 
beholden to another project about our SQLA upgrade timeframe.


So I think that generalities aside, if are a library, and use SQLA, we 
probably need to really think about putting it in the integrated gate.


Because otherwise what we are saying is taskflow is completely dictating 
the SQLA version in the Icehouse release of OpenStack. And that's the 
wrong place for that decision to be.


If taskflow worked with a storage callback mechanism, and got a storage 
interface from the program that was calling it, then things would be 
much different. But because it's going straight to the database and 
managing tables directly, through a known unstable library, that 
OpenStack itself needs some control over, it's definitely a different case.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Joshua Harlow]

Since the library model is what most everyone else uses outside of
openstack (I assume?) what can we do to get there so that this model works
as well?

Expanding dependencies recursively seems like it could help? This could
then detect transitive dependency issues (and doesn't seem so hard to do).

I like the idea of 'gate on all the things' (that I've heard come up
before) but I don't know if its possible? If we hooked into the pypi
upload 'stream' it would seem like we could automatically trigger
openstack builds when a known dependency (or dependency of a
dependency...) is uploaded (maybe?). That would be pretty neat.

In general it really seems like having more libraries, not less is ideal
(making us solve this issue whether we want to or not really). As
libraries can then be used outside of openstack (taskflow is already being
used elsewhere also), libraries create well-defined apis and boundaries
between programs (...). I know they also create this dependency hell
problem (anvil has been hitting these same issues for a while to). But I
think we can figure out a solution that fits both worlds, the world of
things we can gate on and the world of things we can't (3rd party
libraries that aren't under openstack control). Taskflow is in-between
those worlds, so it allows us to explore how to make both of those worlds
work.

-Josh

On 1/3/14, 2:54 PM, "Sean Dague"  wrote:

>On 01/03/2014 05:10 PM, Ivan Melnikov wrote:
>> On 04.01.2014 01:29, Sean Dague wrote:
>>> On 01/03/2014 04:17 PM, Doug Hellmann wrote:
>> [...]
 That's what made me think of the solution. But isn't setuptools in
fact
 telling us that somehow the versions of things we expected to have
 installed are no longer installed and so something *is* broken (even
if
 the versions of the installed libraries work together).
>>>
>>> It actually tells us that a human, somewhere, decided that their
>>> software did not work with some combination of other software, and that
>>> we are no longer able to correct their mistaken assumptions.
>> [...]
>>
>> But sometimes humans are not wrong. For example, no released TaskFlow
>> version really works with SQLAlchemy >= 0.8 -- that was fixed only
>> recently (https://review.openstack.org/#/c/62661/).
>>
>> I consider requirements to be part of the code, so if they are too
>> strict (or too broad), that must be fixed, in a usual opensource way:
>> via filing bug report, suggesting patch and so on.
>>
>> Requirements should reflect reality, especially for libraries that are
>> intended to be useful outside of OpenStack, too. For example, current
>> TaskFlow requirement for SQLAlchemy is too strict, so we'll fix that,
>> and release new version with that fix.
>
>Well part of the problem is because of it being a dependency of a
>dependency, our global requirements process couldn't explore whether or
>not it functioned in a real environment.
>
>Which brings us back to the idea of making this a project that works in
>the integrated gate.
>
>It's also kind of problematic that apparently the introduction of
>taskflow actually caused a regression from Havana (which the distros
>managed to make work with sqla 0.8 even though it wasn't in global
>requirements, but this apparently would have broken).
>
>And the next question is when is a sqla 0.9 compatible version going to
>be out there? Because we can't even explore allowing openstack to use
>sqla 0.9 until taskflow does, again because it's a blocking requirement.
>
>It's exactly these kind of lock step problems that we avoid with the
>rest of openstack by making it an integrated gate with global
>requirements sync. But that we can't really handle with the library model.
>
>   -Sean
>
>-- 
>Sean Dague
>Samsung Research America
>s...@dague.net / sean.da...@samsung.com
>http://dague.net
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

On 01/03/2014 05:10 PM, Ivan Melnikov wrote:

On 04.01.2014 01:29, Sean Dague wrote:

On 01/03/2014 04:17 PM, Doug Hellmann wrote:

[...]

That's what made me think of the solution. But isn't setuptools in fact
telling us that somehow the versions of things we expected to have
installed are no longer installed and so something *is* broken (even if
the versions of the installed libraries work together).


It actually tells us that a human, somewhere, decided that their
software did not work with some combination of other software, and that
we are no longer able to correct their mistaken assumptions.

[...]

But sometimes humans are not wrong. For example, no released TaskFlow
version really works with SQLAlchemy >= 0.8 -- that was fixed only
recently (https://review.openstack.org/#/c/62661/).

I consider requirements to be part of the code, so if they are too
strict (or too broad), that must be fixed, in a usual opensource way:
via filing bug report, suggesting patch and so on.

Requirements should reflect reality, especially for libraries that are
intended to be useful outside of OpenStack, too. For example, current
TaskFlow requirement for SQLAlchemy is too strict, so we'll fix that,
and release new version with that fix.


Well part of the problem is because of it being a dependency of a 
dependency, our global requirements process couldn't explore whether or 
not it functioned in a real environment.


Which brings us back to the idea of making this a project that works in 
the integrated gate.


It's also kind of problematic that apparently the introduction of 
taskflow actually caused a regression from Havana (which the distros 
managed to make work with sqla 0.8 even though it wasn't in global 
requirements, but this apparently would have broken).


And the next question is when is a sqla 0.9 compatible version going to 
be out there? Because we can't even explore allowing openstack to use 
sqla 0.9 until taskflow does, again because it's a blocking requirement.


It's exactly these kind of lock step problems that we avoid with the 
rest of openstack by making it an integrated gate with global 
requirements sync. But that we can't really handle with the library model.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Ivan Melnikov]

On 04.01.2014 01:29, Sean Dague wrote:
> On 01/03/2014 04:17 PM, Doug Hellmann wrote:
[...]
>> That's what made me think of the solution. But isn't setuptools in fact
>> telling us that somehow the versions of things we expected to have
>> installed are no longer installed and so something *is* broken (even if
>> the versions of the installed libraries work together).
> 
> It actually tells us that a human, somewhere, decided that their
> software did not work with some combination of other software, and that
> we are no longer able to correct their mistaken assumptions.
[...]

But sometimes humans are not wrong. For example, no released TaskFlow
version really works with SQLAlchemy >= 0.8 -- that was fixed only
recently (https://review.openstack.org/#/c/62661/).

I consider requirements to be part of the code, so if they are too
strict (or too broad), that must be fixed, in a usual opensource way:
via filing bug report, suggesting patch and so on.

Requirements should reflect reality, especially for libraries that are
intended to be useful outside of OpenStack, too. For example, current
TaskFlow requirement for SQLAlchemy is too strict, so we'll fix that,
and release new version with that fix.

-- 
WBR,
Ivan A. Melnikov

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

On 01/03/2014 04:17 PM, Doug Hellmann wrote:




On Fri, Jan 3, 2014 at 4:08 PM, Sean Dague mailto:s...@dague.net>> wrote:

On 01/03/2014 03:30 PM, Doug Hellmann wrote:




On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague mailto:s...@dague.net>
>> wrote:

 On 01/03/2014 02:44 PM, Doug Hellmann wrote:




 On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow
 mailto:harlo...@yahoo-inc.com>
__>
 

 __>__>> wrote:

  Ok, I think I'm fine with that (although not
really sure
 what that
  entails).

  What does the living under the 'oslo program' change?

  Does that entail getting sucked into the incubator
(which
 seems to
  be what
  your graduating link is about).

  I don't think its a good idea for taskflow to be
in the
 'incubator'.
  Taskflow is meant to be just like any other 3rd
party library.


 No, as we discussed in Hong Kong, there's no reason to add
 taskflow to
 the incubator.

 Whether or not it needs to be part of the oslo program
(or any other
 program) is a separate question. I'm not opposed to
bringing it
 in, but
 didn't see the point when it came up at the summit.

 I understand that moving taskflow into oslo would avoid
the policy
 decision we have in place to not do symmetric gating on
unreleased
 versions of things not "owned" by the OpenStack
project. However, I
 don't know if we want to be testing against the git head of
 libraries no
 matter where they live. As fungi pointed out on IRC,
gating against
 pre-release versions of libraries may allow us to reach
a state
 where
 the software works when installed from git, but not
from the
 released
 packages.

 It seems safer to gate changes to libraries against the
apps'
 trunk (to
 avoid making backwards-incompatible changes), and then gate
 changes to
 the apps against the released libraries (to ensure they
work with
 something available to be packaged by the distros).
There are
 lots and
 lots of version numbers available to us, so I see no
problem with
 releasing new versions of libraries frequently.

 Am I missing something that makes that not work?


 Requirements wedging.

 Because of entry points any library that specifies any
dependencies
 that OpenStack components specify, at incompatible levels,
means
 that library effectively puts a hold on the rest of
OpenStack and
 prevents it from being able to move forward.


I have considered changing stevedore so it uses entry points to find
plugins, but then handles the import itself specifically to
eliminate
the requirements checks for plugin dependencies enforced by
pkg_resources. So far I haven't convinced myself that it's a
good idea,
but I'm open to discussion. :-)


Well, about 1/3 of our requirements wedges are actually completely
entry point related. They were a class of problems we never had
before we switched to entry points.


That's what made me think of the solution. But isn't setuptools in fact
telling us that somehow the versions of things we expected to have
installed are no longer installed and so something *is* broken (even if
the versions of the installed libraries work together).


It actually tells us that a human, somewhere, decided that their 
software did not work with some combination of other software, and that 
we are no longer able to correct their mistaken assumptions.


There is a reason the Linux distros don't blindly follow what libraries 
say they depend on, because humans are foulable, and can only look 
backwards and not forwards in time.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/l

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Doug Hellmann]

On Fri, Jan 3, 2014 at 4:08 PM, Sean Dague  wrote:

> On 01/03/2014 03:30 PM, Doug Hellmann wrote:
>
>>
>>
>>
>> On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague > > wrote:
>>
>> On 01/03/2014 02:44 PM, Doug Hellmann wrote:
>>
>>
>>
>>
>> On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow
>> mailto:harlo...@yahoo-inc.com>
>> >
>> __>> wrote:
>>
>>  Ok, I think I'm fine with that (although not really sure
>> what that
>>  entails).
>>
>>  What does the living under the 'oslo program' change?
>>
>>  Does that entail getting sucked into the incubator (which
>> seems to
>>  be what
>>  your graduating link is about).
>>
>>  I don't think its a good idea for taskflow to be in the
>> 'incubator'.
>>  Taskflow is meant to be just like any other 3rd party
>> library.
>>
>>
>> No, as we discussed in Hong Kong, there's no reason to add
>> taskflow to
>> the incubator.
>>
>> Whether or not it needs to be part of the oslo program (or any
>> other
>> program) is a separate question. I'm not opposed to bringing it
>> in, but
>> didn't see the point when it came up at the summit.
>>
>> I understand that moving taskflow into oslo would avoid the policy
>> decision we have in place to not do symmetric gating on unreleased
>> versions of things not "owned" by the OpenStack project. However,
>> I
>> don't know if we want to be testing against the git head of
>> libraries no
>> matter where they live. As fungi pointed out on IRC, gating
>> against
>> pre-release versions of libraries may allow us to reach a state
>> where
>> the software works when installed from git, but not from the
>> released
>> packages.
>>
>> It seems safer to gate changes to libraries against the apps'
>> trunk (to
>> avoid making backwards-incompatible changes), and then gate
>> changes to
>> the apps against the released libraries (to ensure they work with
>> something available to be packaged by the distros). There are
>> lots and
>> lots of version numbers available to us, so I see no problem with
>> releasing new versions of libraries frequently.
>>
>> Am I missing something that makes that not work?
>>
>>
>> Requirements wedging.
>>
>> Because of entry points any library that specifies any dependencies
>> that OpenStack components specify, at incompatible levels, means
>> that library effectively puts a hold on the rest of OpenStack and
>> prevents it from being able to move forward.
>>
>>
>> I have considered changing stevedore so it uses entry points to find
>> plugins, but then handles the import itself specifically to eliminate
>> the requirements checks for plugin dependencies enforced by
>> pkg_resources. So far I haven't convinced myself that it's a good idea,
>> but I'm open to discussion. :-)
>>
>
> Well, about 1/3 of our requirements wedges are actually completely entry
> point related. They were a class of problems we never had before we
> switched to entry points.
>

That's what made me think of the solution. But isn't setuptools in fact
telling us that somehow the versions of things we expected to have
installed are no longer installed and so something *is* broken (even if the
versions of the installed libraries work together).

Doug



>
> This entire taskflow block wouldn't have been a problem if it wasn't
> loaded with entry points.


>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

On 01/03/2014 03:30 PM, Doug Hellmann wrote:




On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague mailto:s...@dague.net>> wrote:

On 01/03/2014 02:44 PM, Doug Hellmann wrote:




On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow
mailto:harlo...@yahoo-inc.com>
__>> wrote:

 Ok, I think I'm fine with that (although not really sure
what that
 entails).

 What does the living under the 'oslo program' change?

 Does that entail getting sucked into the incubator (which
seems to
 be what
 your graduating link is about).

 I don't think its a good idea for taskflow to be in the
'incubator'.
 Taskflow is meant to be just like any other 3rd party library.


No, as we discussed in Hong Kong, there's no reason to add
taskflow to
the incubator.

Whether or not it needs to be part of the oslo program (or any other
program) is a separate question. I'm not opposed to bringing it
in, but
didn't see the point when it came up at the summit.

I understand that moving taskflow into oslo would avoid the policy
decision we have in place to not do symmetric gating on unreleased
versions of things not "owned" by the OpenStack project. However, I
don't know if we want to be testing against the git head of
libraries no
matter where they live. As fungi pointed out on IRC, gating against
pre-release versions of libraries may allow us to reach a state
where
the software works when installed from git, but not from the
released
packages.

It seems safer to gate changes to libraries against the apps'
trunk (to
avoid making backwards-incompatible changes), and then gate
changes to
the apps against the released libraries (to ensure they work with
something available to be packaged by the distros). There are
lots and
lots of version numbers available to us, so I see no problem with
releasing new versions of libraries frequently.

Am I missing something that makes that not work?


Requirements wedging.

Because of entry points any library that specifies any dependencies
that OpenStack components specify, at incompatible levels, means
that library effectively puts a hold on the rest of OpenStack and
prevents it from being able to move forward.


I have considered changing stevedore so it uses entry points to find
plugins, but then handles the import itself specifically to eliminate
the requirements checks for plugin dependencies enforced by
pkg_resources. So far I haven't convinced myself that it's a good idea,
but I'm open to discussion. :-)


Well, about 1/3 of our requirements wedges are actually completely entry 
point related. They were a class of problems we never had before we 
switched to entry points.


This entire taskflow block wouldn't have been a problem if it wasn't 
loaded with entry points.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Doug Hellmann]

On Fri, Jan 3, 2014 at 3:13 PM, Sean Dague  wrote:

> On 01/03/2014 02:44 PM, Doug Hellmann wrote:
>
>>
>>
>>
>> On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow > > wrote:
>>
>> Ok, I think I'm fine with that (although not really sure what that
>> entails).
>>
>> What does the living under the 'oslo program' change?
>>
>> Does that entail getting sucked into the incubator (which seems to
>> be what
>> your graduating link is about).
>>
>> I don't think its a good idea for taskflow to be in the 'incubator'.
>> Taskflow is meant to be just like any other 3rd party library.
>>
>>
>> No, as we discussed in Hong Kong, there's no reason to add taskflow to
>> the incubator.
>>
>> Whether or not it needs to be part of the oslo program (or any other
>> program) is a separate question. I'm not opposed to bringing it in, but
>> didn't see the point when it came up at the summit.
>>
>> I understand that moving taskflow into oslo would avoid the policy
>> decision we have in place to not do symmetric gating on unreleased
>> versions of things not "owned" by the OpenStack project. However, I
>> don't know if we want to be testing against the git head of libraries no
>> matter where they live. As fungi pointed out on IRC, gating against
>> pre-release versions of libraries may allow us to reach a state where
>> the software works when installed from git, but not from the released
>> packages.
>>
>> It seems safer to gate changes to libraries against the apps' trunk (to
>> avoid making backwards-incompatible changes), and then gate changes to
>> the apps against the released libraries (to ensure they work with
>> something available to be packaged by the distros). There are lots and
>> lots of version numbers available to us, so I see no problem with
>> releasing new versions of libraries frequently.
>>
>> Am I missing something that makes that not work?
>>
>
> Requirements wedging.
>
> Because of entry points any library that specifies any dependencies that
> OpenStack components specify, at incompatible levels, means that library
> effectively puts a hold on the rest of OpenStack and prevents it from being
> able to move forward.
>

I have considered changing stevedore so it uses entry points to find
plugins, but then handles the import itself specifically to eliminate the
requirements checks for plugin dependencies enforced by pkg_resources. So
far I haven't convinced myself that it's a good idea, but I'm open to
discussion. :-)



>
> Which means every time we need to then change a library dependency it's
> going to require triggering a release, solely to change library
> dependencies.
>
> This is the giant disaster that we created global requirements to avoid,
> so that we have one nob to turn.
>
> The only other option is that libraries we own (stackforge / oslo), for
> condition to be included in global- requirements, *can never* specify a
> maximum version of any dependency (and I really mean never), and can never
> specify a minimum greater than current global requirements.


That seems like a more appealing solution than testing against unreleased
versions via git checkouts. It is is also an approach we could encourage
authors of libraries we don't own to use.

Doug



>
>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Jeremy Stanley]

On 2014-01-03 14:44:40 -0500 (-0500), Doug Hellmann wrote:
[...]
> It seems safer to gate changes to libraries against the apps'
> trunk (to avoid making backwards-incompatible changes), and then
> gate changes to the apps against the released libraries (to
> ensure they work with something available to be packaged by the
> distros). There are lots and lots of version numbers available to
> us, so I see no problem with releasing new versions of libraries
> frequently.
[...]

Well, for the benefit of the "pure OpenStack CD" crowd, it would
continue to make sense to also test trunk of everything together. I
think what we had arrived at during the last summit was that we
wanted to make sure that server changes are being gated against
released libraries/clients *in addition to trunk* so that we ensure
features are released in those before we start relying on them. We
similarly probably need to test changes for a library/client against
released versions of any other libraries or clients on which they
depend as well, for those same reasons.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

On 01/03/2014 02:44 PM, Doug Hellmann wrote:




On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow mailto:harlo...@yahoo-inc.com>> wrote:

Ok, I think I'm fine with that (although not really sure what that
entails).

What does the living under the 'oslo program' change?

Does that entail getting sucked into the incubator (which seems to
be what
your graduating link is about).

I don't think its a good idea for taskflow to be in the 'incubator'.
Taskflow is meant to be just like any other 3rd party library.


No, as we discussed in Hong Kong, there's no reason to add taskflow to
the incubator.

Whether or not it needs to be part of the oslo program (or any other
program) is a separate question. I'm not opposed to bringing it in, but
didn't see the point when it came up at the summit.

I understand that moving taskflow into oslo would avoid the policy
decision we have in place to not do symmetric gating on unreleased
versions of things not "owned" by the OpenStack project. However, I
don't know if we want to be testing against the git head of libraries no
matter where they live. As fungi pointed out on IRC, gating against
pre-release versions of libraries may allow us to reach a state where
the software works when installed from git, but not from the released
packages.

It seems safer to gate changes to libraries against the apps' trunk (to
avoid making backwards-incompatible changes), and then gate changes to
the apps against the released libraries (to ensure they work with
something available to be packaged by the distros). There are lots and
lots of version numbers available to us, so I see no problem with
releasing new versions of libraries frequently.

Am I missing something that makes that not work?


Requirements wedging.

Because of entry points any library that specifies any dependencies that 
OpenStack components specify, at incompatible levels, means that library 
effectively puts a hold on the rest of OpenStack and prevents it from 
being able to move forward.


Which means every time we need to then change a library dependency it's 
going to require triggering a release, solely to change library 
dependencies.


This is the giant disaster that we created global requirements to avoid, 
so that we have one nob to turn.


The only other option is that libraries we own (stackforge / oslo), for 
condition to be included in global- requirements, *can never* specify a 
maximum version of any dependency (and I really mean never), and can 
never specify a minimum greater than current global requirements.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Joshua Harlow]

I'd be up for that 'dual' gating. It would help make sure that nothing major is 
breaking in the next version as well as the version released on pypi isn't also 
causing problems.

Although git head gating does seem odd, especially as git head/trunk is where 
things change (and should be allowed to break and change, change is good) and 
the level of false positives that would be raised might become to much of a 
pain. I'd rather not discourage innovation on trunk if we can; since this is 
what stable releases are for.

Btw the sqlalchemy changes (unpinning) should be fine.

Ivan did a couple of tests (basic stuff, not full integration).

- https://review.openstack.org/#/c/64869/
- https://review.openstack.org/#/c/64881/

Both passed basic unit tests (full integration against real mysql, not sqlite 
will happen soon I hope with https://bugs.launchpad.net/taskflow/+bug/1265886).

If we really need to I can push out a 0.1.2 with the unpinned version (one of 
the above reviews).

-Josh

From: Doug Hellmann 
mailto:doug.hellm...@dreamhost.com>>
Date: Friday, January 3, 2014 at 11:44 AM
To: Joshua Harlow mailto:harlo...@yahoo-inc.com>>
Cc: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>, 
Sean Dague mailto:s...@dague.net>>
Subject: Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 
upgrade




On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow 
mailto:harlo...@yahoo-inc.com>> wrote:
Ok, I think I'm fine with that (although not really sure what that
entails).

What does the living under the 'oslo program' change?

Does that entail getting sucked into the incubator (which seems to be what
your graduating link is about).

I don't think its a good idea for taskflow to be in the 'incubator'.
Taskflow is meant to be just like any other 3rd party library.

No, as we discussed in Hong Kong, there's no reason to add taskflow to the 
incubator.

Whether or not it needs to be part of the oslo program (or any other program) 
is a separate question. I'm not opposed to bringing it in, but didn't see the 
point when it came up at the summit.

I understand that moving taskflow into oslo would avoid the policy decision we 
have in place to not do symmetric gating on unreleased versions of things not 
"owned" by the OpenStack project. However, I don't know if we want to be 
testing against the git head of libraries no matter where they live. As fungi 
pointed out on IRC, gating against pre-release versions of libraries may allow 
us to reach a state where the software works when installed from git, but not 
from the released packages.

It seems safer to gate changes to libraries against the apps' trunk (to avoid 
making backwards-incompatible changes), and then gate changes to the apps 
against the released libraries (to ensure they work with something available to 
be packaged by the distros). There are lots and lots of version numbers 
available to us, so I see no problem with releasing new versions of libraries 
frequently.

Am I missing something that makes that not work?

Doug



Or were u mainly referring to the 'devstack-gate integration' section?

I'd be interested in hearing dougs opinion here (cc'd him) as
https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator
would seem to cause even more of these types of new 3rd party libraries to
appear on pypi (and therefore causing similar issues of transitive
dependencies as taskflow).

Will bug u on #openstack-infra soon :-)

On 1/3/14, 9:05 AM, "Sean Dague" mailto:s...@dague.net>> wrote:

>On 01/03/2014 11:37 AM, Joshua Harlow wrote:
>> So taskflow was tested with the version of sqlalchemy that was available
>> and in the requirements at the time of its 0.1 release (taskflow syncs
>> it's requirements from the same global requirements). From what I
>> remember this is the same requirement that everyone else is bound to:
>>
>> SQLAlchemy>=0.7.8,<=0.7.99
>>
>> I can unpin it if this is desired (the openstack requirements repo has
>> the same version restriction). What would be recommended here? As more
>> code moves to pypi reusable libraries (oslo.db when it arrives comes to
>> mind) I think this will be hit more often. Let's come up with a good
>> strategy to follow.
>>
>> Thoughts??
>
>So I think that given taskflow's usage, it really needs to live under
>the Oslo program, and follow the same rules that are applied to oslo
>libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation)
>
>Which means it needs to be part of the integrated gate, so we can update
>it's requirements globally. It also means that changes to it will be
>gated on full devstack runs.
>
>We can work through the details on #openstack-infra. ttx has been doing
>the same for oslo.rootwrap this week.
>
>   -Sean
>
>--
>Sean Dague
>Samsung Research America
>s...@dague.net / 
>sean.da...@samsung.com
>http://dague.net
>
>___
>

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Doug Hellmann]

On Fri, Jan 3, 2014 at 12:45 PM, Joshua Harlow wrote:

> Ok, I think I'm fine with that (although not really sure what that
> entails).
>
> What does the living under the 'oslo program' change?
>
> Does that entail getting sucked into the incubator (which seems to be what
> your graduating link is about).
>
> I don't think its a good idea for taskflow to be in the 'incubator'.
> Taskflow is meant to be just like any other 3rd party library.
>

No, as we discussed in Hong Kong, there's no reason to add taskflow to the
incubator.

Whether or not it needs to be part of the oslo program (or any other
program) is a separate question. I'm not opposed to bringing it in, but
didn't see the point when it came up at the summit.

I understand that moving taskflow into oslo would avoid the policy decision
we have in place to not do symmetric gating on unreleased versions of
things not "owned" by the OpenStack project. However, I don't know if we
want to be testing against the git head of libraries no matter where they
live. As fungi pointed out on IRC, gating against pre-release versions of
libraries may allow us to reach a state where the software works when
installed from git, but not from the released packages.

It seems safer to gate changes to libraries against the apps' trunk (to
avoid making backwards-incompatible changes), and then gate changes to the
apps against the released libraries (to ensure they work with something
available to be packaged by the distros). There are lots and lots of
version numbers available to us, so I see no problem with releasing new
versions of libraries frequently.

Am I missing something that makes that not work?

Doug



>
> Or were u mainly referring to the 'devstack-gate integration' section?
>
> I'd be interested in hearing dougs opinion here (cc'd him) as
> https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator
> would seem to cause even more of these types of new 3rd party libraries to
> appear on pypi (and therefore causing similar issues of transitive
> dependencies as taskflow).
>
> Will bug u on #openstack-infra soon :-)
>
> On 1/3/14, 9:05 AM, "Sean Dague"  wrote:
>
> >On 01/03/2014 11:37 AM, Joshua Harlow wrote:
> >> So taskflow was tested with the version of sqlalchemy that was available
> >> and in the requirements at the time of its 0.1 release (taskflow syncs
> >> it's requirements from the same global requirements). From what I
> >> remember this is the same requirement that everyone else is bound to:
> >>
> >> SQLAlchemy>=0.7.8,<=0.7.99
> >>
> >> I can unpin it if this is desired (the openstack requirements repo has
> >> the same version restriction). What would be recommended here? As more
> >> code moves to pypi reusable libraries (oslo.db when it arrives comes to
> >> mind) I think this will be hit more often. Let's come up with a good
> >> strategy to follow.
> >>
> >> Thoughts??
> >
> >So I think that given taskflow's usage, it really needs to live under
> >the Oslo program, and follow the same rules that are applied to oslo
> >libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation)
> >
> >Which means it needs to be part of the integrated gate, so we can update
> >it's requirements globally. It also means that changes to it will be
> >gated on full devstack runs.
> >
> >We can work through the details on #openstack-infra. ttx has been doing
> >the same for oslo.rootwrap this week.
> >
> >   -Sean
> >
> >--
> >Sean Dague
> >Samsung Research America
> >s...@dague.net / sean.da...@samsung.com
> >http://dague.net
> >
> >___
> >OpenStack-dev mailing list
> >OpenStack-dev@lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Joshua Harlow]

Sounds good to me. 

Talked on #openstack-infra with some folks there and just awaiting next
steps.

Doesn't seem like should be anything to hard to adjust/move/...

-Josh

On 1/3/14, 11:27 AM, "Sean Dague"  wrote:

>On 01/03/2014 12:45 PM, Joshua Harlow wrote:
>> Ok, I think I'm fine with that (although not really sure what that
>> entails).
>>
>> What does the living under the 'oslo program' change?
>>
>> Does that entail getting sucked into the incubator (which seems to be
>>what
>> your graduating link is about).
>>
>> I don't think its a good idea for taskflow to be in the 'incubator'.
>> Taskflow is meant to be just like any other 3rd party library.
>
>I didn't mean the incubator, I meant like oslo.* libs that we've spun
>out already.
>
>> Or were u mainly referring to the 'devstack-gate integration' section?
>
>Correct. Just to understand what the libs live under. Basically taskflow
>is getting deeply integrated into projects in the same way oslo.* libs
>are, and as such, given it has non trivial requirements of it's own, we
>have to treat it like all the other OpenStack components and
>symmetrically gate on it.
>
>That will guaruntee you can't release a taskflow library that can break
>OpenStack, because we'll be testing every commit, which is goodness.
>
>> I'd be interested in hearing dougs opinion here (cc'd him) as
>> https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator
>> would seem to cause even more of these types of new 3rd party libraries
>>to
>> appear on pypi (and therefore causing similar issues of transitive
>> dependencies as taskflow).
>>
>> Will bug u on #openstack-infra soon :-)
>
>Definitely think doug should weigh in as well.
>
>   -Sean
>
>-- 
>Sean Dague
>Samsung Research America
>s...@dague.net / sean.da...@samsung.com
>http://dague.net
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Dev] IDE extensions in .gitignore

[Jeremy Stanley]

On 2014-01-03 19:10:28 + (+), Sam Harwell wrote:
> OpenStack does not have operational or administrative ownership
> over the computers used by contributors. As such, the community
> should not accept or promote any policy which suggests a
> configuration that alters the behavior of systems beyond the scope
> of a local workspace used while working with OpenStack project(s).
> Official alterations of a *global* .gitignore are completely
> unacceptable, but if certain files are not to be specified in the
> .gitignore committed with the project then a policy related to
> modifying the $GIT_DIR/info/exclude would be an acceptable
> alternative.

I really don't understand the aversion to allowing contributors to
police on their own what files they do and don't commit in a review
to an OpenStack project. It all boils down to the following
balancing act:

 * Reviewing changes to each project's .gitignore for the trashfile
   patterns of every editor and IDE known to man is a waste of
   reviewers' collective time.

 * Having to point out to contributors that they've accidentally
   added trashfiles created by their arbitrary choice of tools to a
   change in review is also a waste of reviewers' collective time.

Since there are ways for a contributor to configure their
development environment in a manner which prevents them from
inadvertently putting these files into a change for review, I feel
like it's perfectly reasonable to suggest that as an alternative. It
is just one of the many ways a contributor avoids wasting reviewer
time by neither polluting their changes nor every project's
.gitignore with details potentially relevant only to their own
personal development system and nowhere else.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

On 01/03/2014 12:45 PM, Joshua Harlow wrote:

Ok, I think I'm fine with that (although not really sure what that
entails).

What does the living under the 'oslo program' change?

Does that entail getting sucked into the incubator (which seems to be what
your graduating link is about).

I don't think its a good idea for taskflow to be in the 'incubator'.
Taskflow is meant to be just like any other 3rd party library.


I didn't mean the incubator, I meant like oslo.* libs that we've spun 
out already.



Or were u mainly referring to the 'devstack-gate integration' section?


Correct. Just to understand what the libs live under. Basically taskflow 
is getting deeply integrated into projects in the same way oslo.* libs 
are, and as such, given it has non trivial requirements of it's own, we 
have to treat it like all the other OpenStack components and 
symmetrically gate on it.


That will guaruntee you can't release a taskflow library that can break 
OpenStack, because we'll be testing every commit, which is goodness.



I'd be interested in hearing dougs opinion here (cc'd him) as
https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator
would seem to cause even more of these types of new 3rd party libraries to
appear on pypi (and therefore causing similar issues of transitive
dependencies as taskflow).

Will bug u on #openstack-infra soon :-)


Definitely think doug should weigh in as well.

-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] [devstack-gate] Nominating Sean Dague for devstack-gate-core

[Jeremy Stanley]

On 2014-01-03 09:46:54 -0800 (-0800), James E. Blair wrote:
[...]
> I propose that we create a devstack-gate-core group, and to it we
> add the core infrastructure team and Sean.

I am enthusiastically, wholeheartedly in favor of this proposal.
Sean's reviews and improvements for devstack-gate have been
invaluable.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra] Unit tests, gating, and real databases

[Ivan Melnikov]

Hi there,

As far as I understand, slaves that run gate-*-python27 and python26
jobs have MySQL and Postgres servers installed and running so we can
test migrations and do functional testing for database-related code.
I wanted to use this to improve TaskFlow gating, but I failed to find
docs about it and to derive how this database instances should be
used from nova and oslo.db tests code.

Can anyone give some hints or pointers on where should I get
connection config and what can I do with those database servers in
unit and functional tests?

-- 
WBR,
Ivan A. Melnikov

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra] [devstack-gate] Nominating Sean Dague for devstack-gate-core

[Clark Boylan]

On Fri, Jan 3, 2014 at 9:46 AM, James E. Blair  wrote:
> Occasionally it becomes clear that a part of the project infrastructure
> has its own community interested in it.  Such is the case with
> devstack-gate, which is the nexus of infra and openstack.  Not only does
> it interact with infrastructure systems (in a surprisingly complex way)
> to prepare an integration test run, but it also is very much concerned
> with how those tests are run -- which components are enabled under what
> circumstances, and what configuration they use.
>
> For some time, Sean Dague has shown that not only does he understand the
> mechanics of how the integration test operates, but what the project
> hopes to accomplish by running such tests.  Both the patches he has
> submitted and his reviews indicate a comprehensive knowledge and desire
> to help maintain and improve the system over time.
>
> Therefore I propose that we create a devstack-gate-core group, and to it
> we add the core infrastructure team and Sean.
>
> Thanks,
>
> Jim
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

++

Clark

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack-Dev] IDE extensions in .gitignore

[Sam Harwell]

OpenStack does not have operational or administrative ownership over the 
computers used by contributors. As such, the community should not accept or 
promote any policy which suggests a configuration that alters the behavior of 
systems beyond the scope of a local workspace used while working with OpenStack 
project(s). Official alterations of a *global* .gitignore are completely 
unacceptable, but if certain files are not to be specified in the .gitignore 
committed with the project then a policy related to modifying the 
$GIT_DIR/info/exclude would be an acceptable alternative.

Thanks,
Sam

-Original Message-
From: John Griffith [mailto:john.griff...@solidfire.com] 
Sent: Tuesday, December 31, 2013 10:46 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [OpenStack-Dev] IDE extensions in .gitignore

Hey Everyone,

I wanted to see where we stand on IDE extensions in .gitignore files.
We seem to have some back and forth, one cycle there's a bug and a patch to add 
things like eclipse, idea etc and the next there's a bug and a patch to remove 
them.  I'd like to have some sort of consensus on what we want here.  I 
personally don't have a preference, I would just like to have consistency and 
quit thrashing back and forth.

Anyway, I'd like to see all of the projects agree on this... or even consider 
moving to a global .gitignore.  Thoughts??

John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] neutron plugin for opendaylight

[Vijayendra BVS]

Hi Kyle,

Thanks for the info, I'll try the setup and ping you/others on freenode in
case I need any help.

Regards,
Vijay


On Fri, Jan 3, 2014 at 8:22 AM, Kyle Mestery wrote:

> On Jan 2, 2014, at 7:50 PM, Vijayendra BVS 
> wrote:
> >
> > [Apologies if mesg is duplicated]
> >
> >
> >
> > Hi Kyle/Team,
> >
> >
> >
> > I’m looking at open daylight support in openstack and see the relevant
> blueprint for the neutron plugin for ODL at
> https://docs.google.com/document/d/1rdowsQSYBirS634RMFePOaJu1FysjUUZtobChTkfOIY/edit-
>   is there a repo where we can review the code implemented up until now
> and test it out? I can participate in the implementation effort as well,
> please let me know if there are some modules that need to be addressed.
> >
> >
> Hi Vijay:
>
> The OpenStack integration with OpenDaylight is currently done with an
> ML2 MechanismDriver, and it is not pushed upstream yet. Most of the
> work has been on the OpenDaylight side, as the MechanismDriver is
> very thin. I wrote a blog post on how to take this for a test drive [1], so
> please have a look at that and ping me or the other folks working on
> this on #opendaylight-ovsdb on Freenode.
>
> Thanks,
> Kyle
>
> [1] http://www.siliconloons.com/?p=523
>
> >
> > Regards,
> >
> > Vijay
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[James E. Blair]

Dan Prince  writes:

> - Original Message -
>> From: "Michael Still" 
>> - commenting "recheck .*"
>> - commenting "recheck migrations"
>
> With the growing interest in 3rd party testing systems would using 'recheck 
> turbo-hipster' make more sense here?
>
> I'm fine with 'recheck migrations' in addition for turbo-hipster but it would 
> make sense to align the recheck naming scheme with the title of the reviewer 
> for the 3rd party testing system.

This is the can of worms I was hoping we would not open.  Or try to get
them all back into the can and close it again is perhaps the better
metaphor.

I do not think that system-specific recheck commands will be actually
that useful or important.  As I mentioned, I understand the theoretical
usefulness of being able to say "oops, I can tell this one system messed
up, let's ask it to try again".  But given that case is covered by
asking all systems to recheck, it seems harmless to say "all systems to
recheck".

I just don't think that asking developers to learn a micro-language of
commands left in gerrit comments is the best use of time.  Maybe I'm
wrong about that, but I was hoping we could try not creating it first to
see if there's really a need.  Dealing with the expected errors of a
system first coming online doesn't, in my mind, demonstrate that need.

If it turns out that asking programmers not to create a new language is
futile, yes, I'd rather we have some predictability.  Most third party
CI systems have descriptive names, so rather than saying "recheck
turbo-hipster", perhaps we should change turbo-hipster's display name to
something related to "migrations".

So _if_ we want to have system-specific recheck commands, then I think
we should ask operators to make them in the form "recheck ",
and that they output a brief sentence of help text to remind people of
that in the report message in Gerrit.

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Joshua Harlow]

Ok, I think I'm fine with that (although not really sure what that
entails).

What does the living under the 'oslo program' change?

Does that entail getting sucked into the incubator (which seems to be what
your graduating link is about).

I don't think its a good idea for taskflow to be in the 'incubator'.
Taskflow is meant to be just like any other 3rd party library.

Or were u mainly referring to the 'devstack-gate integration' section?

I'd be interested in hearing dougs opinion here (cc'd him) as
https://etherpad.openstack.org/p/icehouse-oslo-splitting-the-incubator
would seem to cause even more of these types of new 3rd party libraries to
appear on pypi (and therefore causing similar issues of transitive
dependencies as taskflow).

Will bug u on #openstack-infra soon :-)

On 1/3/14, 9:05 AM, "Sean Dague"  wrote:

>On 01/03/2014 11:37 AM, Joshua Harlow wrote:
>> So taskflow was tested with the version of sqlalchemy that was available
>> and in the requirements at the time of its 0.1 release (taskflow syncs
>> it's requirements from the same global requirements). From what I
>> remember this is the same requirement that everyone else is bound to:
>>
>> SQLAlchemy>=0.7.8,<=0.7.99
>>
>> I can unpin it if this is desired (the openstack requirements repo has
>> the same version restriction). What would be recommended here? As more
>> code moves to pypi reusable libraries (oslo.db when it arrives comes to
>> mind) I think this will be hit more often. Let's come up with a good
>> strategy to follow.
>>
>> Thoughts??
>
>So I think that given taskflow's usage, it really needs to live under
>the Oslo program, and follow the same rules that are applied to oslo
>libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation)
>
>Which means it needs to be part of the integrated gate, so we can update
>it's requirements globally. It also means that changes to it will be
>gated on full devstack runs.
>
>We can work through the details on #openstack-infra. ttx has been doing
>the same for oslo.rootwrap this week.
>
>   -Sean
>
>-- 
>Sean Dague
>Samsung Research America
>s...@dague.net / sean.da...@samsung.com
>http://dague.net
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [infra] [devstack-gate] Nominating Sean Dague for devstack-gate-core

[James E. Blair]

Occasionally it becomes clear that a part of the project infrastructure
has its own community interested in it.  Such is the case with
devstack-gate, which is the nexus of infra and openstack.  Not only does
it interact with infrastructure systems (in a surprisingly complex way)
to prepare an integration test run, but it also is very much concerned
with how those tests are run -- which components are enabled under what
circumstances, and what configuration they use.

For some time, Sean Dague has shown that not only does he understand the
mechanics of how the integration test operates, but what the project
hopes to accomplish by running such tests.  Both the patches he has
submitted and his reviews indicate a comprehensive knowledge and desire
to help maintain and improve the system over time.

Therefore I propose that we create a devstack-gate-core group, and to it
we add the core infrastructure team and Sean.

Thanks,

Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Turbo-hipster

[Dan Prince]

- Original Message -
> From: "Michael Still" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Thursday, January 2, 2014 7:48:16 PM
> Subject: Re: [openstack-dev] [nova] Turbo-hipster
> 
> On Fri, Jan 3, 2014 at 9:39 AM, Michael Still  wrote:
> > On Fri, Jan 3, 2014 at 9:24 AM, James E. Blair 
> > wrote:
> >
> >> However, there are _a lot_ of third-party test systems coming on-line,
> >> and I'm not sure that expanding the "recheck language" to support ever
> >> more complexity is a good idea.  I can see how being able to say
> >> "recheck foo" would be useful in some circumstances, but given that just
> >> saying "recheck" will suffice, I'd prefer that we kept the general
> >> recommendation simple so developers can worry about something else.
> >
> > Fair enough. I feel like you and I should sit down and chat about this
> > stuff at the LCA meetup next week. If we need to make tweaks based on
> > that, then we will.
> 
> Further to this, I have just reloaded our zuul with a rules change.
> The following events will all cause a turbo hipster check run:
> 
> - uploading a patchset
> - restoring a patchset
> - commenting "recheck .*"
> - commenting "recheck migrations"

With the growing interest in 3rd party testing systems would using 'recheck 
turbo-hipster' make more sense here?

I'm fine with 'recheck migrations' in addition for turbo-hipster but it would 
make sense to align the recheck naming scheme with the title of the reviewer 
for the 3rd party testing system.


> 
> Cheers,
> Michael
> 
> --
> Rackspace Australia
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [elastic-recheck] Thoughts on next steps

[James E. Blair]

Sean Dague  writes:

> So my feeling is we should move away from the point graphs we have,
> and present these as weekly and daily failure rates (with graphs and
> error bars). And slice those per job. My suggestion is that we do the
> actual visualization with matplotlib because it's super easy to output
> that from pandas data sets.

I am very excited about this and everything above it!

> = Take over of /recheck =
>
> There is still a bunch of useful data coming in on "recheck bug "
> data which hasn't been curated into ER queries. I think the right
> thing to do is treat these as a work queue of bugs we should be
> building patterns out of (or completely invalidating). I've got a
> preliminary gerrit bulk query piece of code that does this, which
> would remove the need of the daemon the way that's currently
> happening. The gerrit queries are a little long right now, but I think
> if we are only doing this on hourly cron, the additional load will be
> negligible.

I think this is fine and am all for reducing complexity, but consider
this alternative: over the break, I moved both components of
elastic-recheck onto a new server (status.openstack.org).  Since they
are now co-located, you could have the component of e-r that watches the
stream to provide responses to gerrit also note recheck actions.  You
could stick the data in a file, memcache, trove database, etc, and the
status page could display that "work queue".  No extra daemons required.

I think the main user-visible aspect of this decision is the delay
before unprocessed bugs are made visible.  If a bug starts affecting a
number of jobs, it might be nice to see what bug numbers people are
using for rechecks without waiting for the next cron run.

On another topic, it's worth mentioning that we now (again, this is new
from over the break) have timeouts _inside_ the devstack-gate jobs that
should hit before the Jenkins timeout, so log collection for
devstack-gate jobs that run long and hit the timeout should still happen
(meaning that e-r can now see these failures).

Thanks for all your work on this.  I think it's extremely useful and
exciting!

-Jim

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

On 01/03/2014 11:37 AM, Joshua Harlow wrote:

So taskflow was tested with the version of sqlalchemy that was available
and in the requirements at the time of its 0.1 release (taskflow syncs
it's requirements from the same global requirements). From what I
remember this is the same requirement that everyone else is bound to:

SQLAlchemy>=0.7.8,<=0.7.99

I can unpin it if this is desired (the openstack requirements repo has
the same version restriction). What would be recommended here? As more
code moves to pypi reusable libraries (oslo.db when it arrives comes to
mind) I think this will be hit more often. Let's come up with a good
strategy to follow.

Thoughts??


So I think that given taskflow's usage, it really needs to live under 
the Oslo program, and follow the same rules that are applied to oslo 
libraries. (https://wiki.openstack.org/wiki/Oslo#Graduation)


Which means it needs to be part of the integrated gate, so we can update 
it's requirements globally. It also means that changes to it will be 
gated on full devstack runs.


We can work through the details on #openstack-infra. ttx has been doing 
the same for oslo.rootwrap this week.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Joshua Harlow]

So taskflow was tested with the version of sqlalchemy that was available and in 
the requirements at the time of its 0.1 release (taskflow syncs it's 
requirements from the same global requirements). From what I remember this is 
the same requirement that everyone else is bound to:

SQLAlchemy>=0.7.8,<=0.7.99

I can unpin it if this is desired (the openstack requirements repo has the same 
version restriction). What would be recommended here? As more code moves to 
pypi reusable libraries (oslo.db when it arrives comes to mind) I think this 
will be hit more often. Let's come up with a good strategy to follow.

Thoughts??

Sent from my really tiny device...

On Jan 3, 2014, at 7:30 AM, "Sean Dague" 
mailto:s...@dague.net>> wrote:

Given that sqla 0.9 just came out, I wanted to explore, again, what the state 
of the world was with sqla 0.8 (especially given that Ubuntu and Red Hat are 
both shipping 0.8 in their OpenStack bundles) - 
https://review.openstack.org/#/c/64831/

The answer is not great. But more importantly, the answer is actually worse 
than the last time we checked, which I think gives this some urgency to move 
forward.

One of the key problems is taskflow, which has an sqla pin, which breaks all 
the cinder entry points. This was actually entirely the problem global 
requirements was meant to address, but it really can't when there are nested 
requirements like this, in stackforge projects that aren't installing via git 
(so we can rewrite their requirements).

So why does taskflow have the SQLA pin? And if the answer is global 
requirements, then taskflow can't be installing via pypi like it is now, 
because it's by nature taking a wedge. So we need a real solution here to un 
bind us, because right now it's actually impossible to upgrade sqla in 
requirements because of taskflow.

   -Sean

--
Sean Dague
Samsung Research America
s...@dague.net / 
sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] minimum review period for functional changes that break backwards compatibility

[David Kranz]

On 01/03/2014 08:52 AM, Thierry Carrez wrote:

Tim Bell wrote:

Is there a mechanism to tag changes as being potentially more appropriate for 
the more ops related profiles ? I'm thinking more when someone proposes a 
change they suspect could have an operations impact, they could highlight this 
as being one for particular focus.

How about an OpsImpact tag ?

I think such a tag would help. That would encourage ops to start looking
more regularly into proposed changes by highlighting the few reviews
that are most likely to need their expertise.

We could have that tag post reviews to the -operators ML (in the same
way SecurityImpact posts to the -security ML), which would additionally
reinforce the need for this list as a separate list from the openstack
general list.

While this would be an improvement over the current situation, IMO we 
are focused a bit too much here on "operators" vs others. I think we 
need clearer guidelines on what an "incompatible change" is, and how to 
balance "change it to something better" with "don't cause users upgrade 
pain". There was a similar issue with API changes a while back and 
providing the api stability guidelines really helped people understand 
the issue better. Of course, similar to what Sean talked about, having 
API tests in tempest that blocked incompatible api changes was probably 
even more important.


 -David



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon] Javascript checkstyle improvement

[Monty Taylor]

On 01/03/2014 03:10 AM, Maxime Vidori wrote:
> Someone said Grunt?

Actually - funny story - we're currently working through some use of
grunt and friends for storyboard, and as a result of that I've been
learning about how it all hangs together. From what I can tell, I think
the path to directly using it in horizon might be a good one to look at
- but it also needs to be done very carefully. Specifically, as you all
know, we do a LOT of work to deal with pip dependencies in the gate. If
we start introducing npm dependencies, we're going to need to do a lot
of infra thinking about how to do that with the same robustness. If we
do though, it could potentially mean we could stop bundling, which would
be neat.

SO -

a) I fully support more jshint for horizon javascript
b) I provisionally support aligning more with current state-of-art
javascript toolchains for the javascript portions of horizon
c) There are MANY unknowns with how to effectively integrate that into a
python-centric workflow.

> - Original Message -
> From: "Matthias Runge" 
> To: openstack-dev@lists.openstack.org
> Sent: Friday, January 3, 2014 9:52:48 AM
> Subject: Re: [openstack-dev] [horizon] Javascript checkstyle improvement
> 
> On 01/03/2014 09:37 AM, Radomir Dopieralski wrote:
> 
>> [snip]
>>
>> This is actually not a problem at all, because the way jshint works now,
>> we have to explicitly list the files to be checked against those
>> rules. That means, that we can only check our own code, and not the
>> included libraries. Of course, the reviewers have to pay attention to
>> make sure that all the non-library code is added to the list, but we
>> don't really get new files that often.
>>
> Yeah, jshint needs to be switched on explicitly for each file. That
> tends to be forgotten on new files, which is not ideal either.
> 
> Matthias
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] neutron plugin for opendaylight

[Kyle Mestery]

On Jan 2, 2014, at 7:50 PM, Vijayendra BVS  wrote:
> 
> [Apologies if mesg is duplicated]
> 
> 
> 
> Hi Kyle/Team,
> 
> 
> 
> I’m looking at open daylight support in openstack and see the relevant 
> blueprint for the neutron plugin for ODL at 
> https://docs.google.com/document/d/1rdowsQSYBirS634RMFePOaJu1FysjUUZtobChTkfOIY/edit
>  -  is there a repo where we can review the code implemented up until now and 
> test it out? I can participate in the implementation effort as well, please 
> let me know if there are some modules that need to be addressed.
> 
> 
Hi Vijay:

The OpenStack integration with OpenDaylight is currently done with an
ML2 MechanismDriver, and it is not pushed upstream yet. Most of the
work has been on the OpenDaylight side, as the MechanismDriver is
very thin. I wrote a blog post on how to take this for a test drive [1], so
please have a look at that and ping me or the other folks working on
this on #opendaylight-ovsdb on Freenode.

Thanks,
Kyle

[1] http://www.siliconloons.com/?p=523

> 
> Regards,
> 
> Vijay
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Vertica Storage Driver Testing

[Herndon, John Luke]

On 1/2/14, 7:06 PM, "Sean Dague"  wrote:

>On 01/02/2014 08:36 PM, Robert Collins wrote:
>> On 3 January 2014 14:34, Robert Collins 
>>wrote:
>>> On 3 January 2014 12:40, Herndon, John Luke 
>>>wrote:


 On 1/2/14, 4:27 PM, "Clint Byrum"  wrote:

>>>
>>>
 I don¹t think it would be that hard to get the review or gate jobs to
use
 a real vertica instance, actually. Who do I talk to about that?
>>>
>>> http://ci.openstack.org/third_party.html
>>
>> Oh, if you meant setting up a gate variant to use vertica community
>> edition - I'd run it past the ceilometer folk and then just submit
>> patches to devstack, devstack-gate and infra/config to do it.
>>
>> devstack - code for setting up a real vertica
>> devstack-gate - handles passing the right flags to devstack for the
>> configuration scenarios we test against
>> infra/config - has the jenkins job builder definitions to define the
>>jobs
>
>I think general policy (thus far) has been that we're not going to put
>non Open Source software into upstream gate jobs.
>
>So you really should approach this via 3rd party testing instead. The
>DB2 folks are approaching it that way, for that reason.

Ok, that makes sense, but tbh, setting up 3rd party testing is going to be
as much or more work than writing the driver. Given schedule constraints,
it probably isn¹t feasible right now. I think for starters, I will write
some unit tests that ensure that changes to the storage interface don¹t
break the driver, and will work on a 3rd party testing strategy over time.

Thanks!
-john


>   -Sean
>
>-- 
>Sean Dague
>Samsung Research America
>s...@dague.net / sean.da...@samsung.com
>http://dague.net
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


smime.p7s
Description: S/MIME cryptographic signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] "import only module" message and #noqa

[Russell Bryant]

On 01/03/2014 10:35 AM, Radomir Dopieralski wrote:
> On 03/01/14 16:18, Russell Bryant wrote:
>> On 01/03/2014 10:10 AM, Radomir Dopieralski wrote:
>>> I think that we can actually do a little bit better and remove many of
>>> the #noqa tags without forfeiting automatic checking. I submitted a
>>> patch: https://review.openstack.org/#/c/64832/
>>>
>>> This basically adds a h302_exceptions option to tox.ini, that lets us
>>> specify which names are allowed to be imported. For example, we can do:
>>>
>>> [hacking]
>>> h302_exceptions = django.conf.settings,
>>>   django.utils.translation.ugettext_lazy,
>>>   django.core.urlresolvers.
>>>
>>> To have settings, _ and everything from urlresolvers importable without
>>> the need for the #noqa tag.
>>>
>>> Of course every project can add their own names there, depending what
>>> they need.
>>
>> Isn't that what "import_exceptions" is for?  For example, we have this
>> in nova:
>>
>> import_exceptions = nova.openstack.common.gettextutils._
>>
> No exactly, as this will disable all import checks, just like # noqa.
> 

Ah, makes sense.  Thanks.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] "import only module" message and #noqa

[Gabriel pettier]

On Fri, Jan 03, 2014 at 10:18:58AM -0500, Russell Bryant wrote:
> On 01/03/2014 10:10 AM, Radomir Dopieralski wrote:
> > I think that we can actually do a little bit better and remove many of
> > the #noqa tags without forfeiting automatic checking. I submitted a
> > patch: https://review.openstack.org/#/c/64832/
> > 
> > This basically adds a h302_exceptions option to tox.ini, that lets us
> > specify which names are allowed to be imported. For example, we can do:
> > 
> > [hacking]
> > h302_exceptions = django.conf.settings,
> >   django.utils.translation.ugettext_lazy,
> >   django.core.urlresolvers.
> > 
> > To have settings, _ and everything from urlresolvers importable without
> > the need for the #noqa tag.
> > 
> > Of course every project can add their own names there, depending what
> > they need.
> 
> Isn't that what "import_exceptions" is for?  For example, we have this
> in nova:
> 
> import_exceptions = nova.openstack.common.gettextutils._
> 
> -- 
> Russell Bryant
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

I guess it could work too, but that would disable all warnings against 
these imports (unused imports, * imports, etc), right? I think a 
specific option is useful.

-- 
Gabriel Pettier
Software Engineer at CloudWatt.com 
06 85 10 36 34

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] "import only module" message and #noqa

[Radomir Dopieralski]

On 03/01/14 16:18, Russell Bryant wrote:
> On 01/03/2014 10:10 AM, Radomir Dopieralski wrote:
>> I think that we can actually do a little bit better and remove many of
>> the #noqa tags without forfeiting automatic checking. I submitted a
>> patch: https://review.openstack.org/#/c/64832/
>>
>> This basically adds a h302_exceptions option to tox.ini, that lets us
>> specify which names are allowed to be imported. For example, we can do:
>>
>> [hacking]
>> h302_exceptions = django.conf.settings,
>>   django.utils.translation.ugettext_lazy,
>>   django.core.urlresolvers.
>>
>> To have settings, _ and everything from urlresolvers importable without
>> the need for the #noqa tag.
>>
>> Of course every project can add their own names there, depending what
>> they need.
> 
> Isn't that what "import_exceptions" is for?  For example, we have this
> in nova:
> 
> import_exceptions = nova.openstack.common.gettextutils._
> 
No exactly, as this will disable all import checks, just like # noqa.

-- 
Radomir Dopieralski

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [requirements] - taskflow preventing sqla 0.8 upgrade

[Sean Dague]

Given that sqla 0.9 just came out, I wanted to explore, again, what the 
state of the world was with sqla 0.8 (especially given that Ubuntu and 
Red Hat are both shipping 0.8 in their OpenStack bundles) - 
https://review.openstack.org/#/c/64831/


The answer is not great. But more importantly, the answer is actually 
worse than the last time we checked, which I think gives this some 
urgency to move forward.


One of the key problems is taskflow, which has an sqla pin, which breaks 
all the cinder entry points. This was actually entirely the problem 
global requirements was meant to address, but it really can't when there 
are nested requirements like this, in stackforge projects that aren't 
installing via git (so we can rewrite their requirements).


So why does taskflow have the SQLA pin? And if the answer is global 
requirements, then taskflow can't be installing via pypi like it is now, 
because it's by nature taking a wedge. So we need a real solution here 
to un bind us, because right now it's actually impossible to upgrade 
sqla in requirements because of taskflow.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [qa] Use test scenarios for interface type and auth version

[Frittoli, Andrea (Cloud Services)]

Hi,

 

I'm working on multi-keystone-api-version-tests which aims to provide the
ability to run tests using a configurable list of keystone API versions for
authentication.

 

If two versions of the API are configured, each test would be run using the
two versions.  

In the gate, where we're short of time, we could test only the latest stable
version, but in other tests will have the ability to use more than one
version.

 

The approach I'm considering is based on testscenarios. 

Thinking about how to implement this I recognized that a solution that works
for auth version could be used also to run json / xml, removing the need to
have two classes for API test.

 

The idea I have is to change Base[Service]Test classes and move the client
selection logic to BaseTestCase.  Something like this:

 

def __getattr__(self, attr):

if attr in
manager.list_of_available_clients:

manager.get_client(attr,
self.auth_version, self.interface)

  else:

 raise AttributeError

Where both auth_version and interface can be injected by testscenarios and
be different for different test instances.

The manager in clients.py could be condensed a bit, using __getattr__ and a
list_of_available_clients, mapped to the corresponding service class.

 

Several tests create a self.client = self.my_favourite_client, which I would
need to remove.

 

As it would be a relatively substantial change, I wanted to get some input /
opinion before I started implementing it.

 

Thanks!

 

andrea

 

 

-- 

Andrea Frittoli

IaaS Systems Engineering Team

HP Cloud Services

Tel: +44(0)11731 62324

 



smime.p7s
Description: S/MIME cryptographic signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] "import only module" message and #noqa

[Russell Bryant]

On 01/03/2014 10:10 AM, Radomir Dopieralski wrote:
> I think that we can actually do a little bit better and remove many of
> the #noqa tags without forfeiting automatic checking. I submitted a
> patch: https://review.openstack.org/#/c/64832/
> 
> This basically adds a h302_exceptions option to tox.ini, that lets us
> specify which names are allowed to be imported. For example, we can do:
> 
> [hacking]
> h302_exceptions = django.conf.settings,
>   django.utils.translation.ugettext_lazy,
>   django.core.urlresolvers.
> 
> To have settings, _ and everything from urlresolvers importable without
> the need for the #noqa tag.
> 
> Of course every project can add their own names there, depending what
> they need.

Isn't that what "import_exceptions" is for?  For example, we have this
in nova:

import_exceptions = nova.openstack.common.gettextutils._

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] "import only module" message and #noqa

[Radomir Dopieralski]

On 30/12/13 16:10, Gabriel pettier wrote:
> So Tatiana pointed 
> http://lists.openstack.org/pipermail/openstack-dev/2013-June/thread.html#9993 
> to me, and from there i went on to read 
> http://lists.openstack.org/pipermail/openstack-dev/2013-August/thread.html#13074
>  
> and i can see valid points for H302, even if it annoy me sometime, if 
> it's better for reviews, i understand.
> 
> Sorry for the noise
> 
> On Mon, Dec 30, 2013 at 03:43:03PM +0100, Gabriel pettier wrote:
>> Hi
>>
>> Reading horizon's code and recent reviews, i'm under the impression that 
>> it's a common practice to use #noqa to bypass the "import only modules" 
>> qa message, i'm unconvinced of the advantages of this policy (i think 
>> the namespace is often cleaner when one import only the symbols needed 
>> from the modules), so i think this policy could be removed, by adding 
>> "H302" to the list of ignored errors in tox.ini.
>>
>> This would allow removing a lot of #noqa comments, making for cleaner 
>> code.
>>
>> If there are significant advantages to this policy, however, it should 
>> be made more consistently applied to fix all these imports.

I think that we can actually do a little bit better and remove many of
the #noqa tags without forfeiting automatic checking. I submitted a
patch: https://review.openstack.org/#/c/64832/

This basically adds a h302_exceptions option to tox.ini, that lets us
specify which names are allowed to be imported. For example, we can do:

[hacking]
h302_exceptions = django.conf.settings,
  django.utils.translation.ugettext_lazy,
  django.core.urlresolvers.

To have settings, _ and everything from urlresolvers importable without
the need for the #noqa tag.

Of course every project can add their own names there, depending what
they need.
-- 
Radomir Dopieralski

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] minimum review period for functional changes that break backwards compatibility

[Sean Dague]

On 01/03/2014 09:44 AM, Anne Gentle wrote:




On Fri, Jan 3, 2014 at 7:52 AM, Thierry Carrez mailto:thie...@openstack.org>> wrote:

Tim Bell wrote:
 > Is there a mechanism to tag changes as being potentially more
appropriate for the more ops related profiles ? I'm thinking more
when someone proposes a change they suspect could have an operations
impact, they could highlight this as being one for particular focus.
 >
 > How about an OpsImpact tag ?

I think such a tag would help. That would encourage ops to start looking
more regularly into proposed changes by highlighting the few reviews
that are most likely to need their expertise.

We could have that tag post reviews to the -operators ML (in the same
way SecurityImpact posts to the -security ML), which would additionally
reinforce the need for this list as a separate list from the openstack
general list.


I think this is a good idea, though I have to say when we were using
DocImpact in this way, it kills conversational dialog on the mailing
list because all the robots start talking on your list as more people
use the flag. We have since moved to automatically logging a doc bug
rather than posting to the mailing list.

Still, I think it's worth trying for OpsImpact because there's not a
secondary place to log the impactful patches. Nice idea, Tim!


If we are going down this path, it would be good to give reviewers a 
training set. So a dozen patches that someone things would be good 
OpsImpact and a dozen patches which clearly aren't.


Otherwise I fear we're making a lot of assumptions about what people 
believe are and are not OpsImpact and I don't think it will be that helpful.


I also really think that this is the less useful approach than taking 
those key ops issues into the CI toolchain. And if people only have one 
place to spend energy, spending it on the CI side is way more beneficial.


-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] minimum review period for functional changes that break backwards compatibility

[Anne Gentle]

On Fri, Jan 3, 2014 at 7:52 AM, Thierry Carrez wrote:

> Tim Bell wrote:
> > Is there a mechanism to tag changes as being potentially more
> appropriate for the more ops related profiles ? I'm thinking more when
> someone proposes a change they suspect could have an operations impact,
> they could highlight this as being one for particular focus.
> >
> > How about an OpsImpact tag ?
>
> I think such a tag would help. That would encourage ops to start looking
> more regularly into proposed changes by highlighting the few reviews
> that are most likely to need their expertise.
>
> We could have that tag post reviews to the -operators ML (in the same
> way SecurityImpact posts to the -security ML), which would additionally
> reinforce the need for this list as a separate list from the openstack
> general list.
>
>
I think this is a good idea, though I have to say when we were using
DocImpact in this way, it kills conversational dialog on the mailing list
because all the robots start talking on your list as more people use the
flag. We have since moved to automatically logging a doc bug rather than
posting to the mailing list.

Still, I think it's worth trying for OpsImpact because there's not a
secondary place to log the impactful patches. Nice idea, Tim!

Anne


>  --
> Thierry Carrez (ttx)
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Anne Gentle
annegen...@justwriteclick.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] [qa] oslo.config generator scrambling tempest config files

[Sean Dague]

On 01/03/2014 08:07 AM, Davanum Srinivas wrote:
> Sean,
> 
> Do you already have this in tempest copy of the code?
> https://review.openstack.org/#/c/60401/ - that will help going
> forward.
> 
> -- dims

We did not, syncing now.

It's sort of a bummer that the code isn't registration order. As we do
multi group registration in a particular sequence for a reason, as it
made logical sense to order groups in a certain way (which may not be
alphabetical).

But barring that, alphabetical is at least stable.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] minimum review period for functional changes that break backwards compatibility

[Thierry Carrez]

Tim Bell wrote:
> Is there a mechanism to tag changes as being potentially more appropriate for 
> the more ops related profiles ? I'm thinking more when someone proposes a 
> change they suspect could have an operations impact, they could highlight 
> this as being one for particular focus.
> 
> How about an OpsImpact tag ?

I think such a tag would help. That would encourage ops to start looking
more regularly into proposed changes by highlighting the few reviews
that are most likely to need their expertise.

We could have that tag post reviews to the -operators ML (in the same
way SecurityImpact posts to the -security ML), which would additionally
reinforce the need for this list as a separate list from the openstack
general list.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Glance] Passing error data in Glance exceptions?

[David Koo]

Hi All,

At the moment the GlanceException class has only a static
"message" member which holds the error string. As a result higher
level modules (e.g. API) are unable to translate a lower level
exception to a more meaningful user-facing message - they either have
to construct a totally new message or reproduce the exception string.

Conversely, lower level modules (e.g. storage backend modules)
need to consider their parameters carefully (e.g. some "reason"
string) if the parameter is some form of free-flowing string (as
against, say, an int value) because the string might be reported to
the end user.

All this seems a little haphazzard.

But if we just tweaked the GlanceException class a little bit to
store the 'args' and 'kwargs' arguments, then the lower level modules
can make their error info available to the higher layers and the
higher layers can do further analysis of the errors reported by the
lower layers and give an appropriate user-facing description of the
error (or simply generate more meaningful logs than just reproducing
what the lower layer already logged).

And I think it will also make testing a little simpler - instead
of testing for assertTrue("spam ham" in exc.message) we can just do
a more precise assertEquals("spam ham", exc.eggs).

What do you all think?

--
Koo

PS: The background to all this is a long exchange we're having in:
https://review.openstack.org/#/c/64409/
I welcome/encourage/request/beg more reviewers to take a look and
chime in their valuable opinions. Many thanks.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] [qa] oslo.config generator scrambling tempest config files

[Davanum Srinivas]

Sean,

Do you already have this in tempest copy of the code?
https://review.openstack.org/#/c/60401/ - that will help going
forward.

-- dims

On Fri, Jan 3, 2014 at 7:53 AM, Sean Dague  wrote:
> So maybe I only just noticed it, but we moved to using the oslo.config
> generator in Tempest, and it's largely succeeding at creating really
> large config diffs and moving sections around all over the place.
>
> Exhibit A: https://review.openstack.org/#/c/60434/14/etc/tempest.conf.sample
>
> Which means it's generating a lot of noise, and I don't really
> understand why. It would seem that a config generator should be order
> stable. Anyone familiar enough with this to know why? And why this
> doesn't seem to be an issue with other projects?
>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: http://davanum.wordpress.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Tuskar CLI after architecture changes

[Jiří Stránský]

On 21.12.2013 06:10, Jay Pipes wrote:

On 12/20/2013 11:34 AM, Clint Byrum wrote:

Excerpts from Radomir Dopieralski's message of 2013-12-20 01:13:20 -0800:

On 20/12/13 00:17, Jay Pipes wrote:

On 12/19/2013 04:55 AM, Radomir Dopieralski wrote:

On 14/12/13 16:51, Jay Pipes wrote:

[snip]


Instead of focusing on locking issues -- which I agree are very
important in the virtualized side of things where resources are
"thinner" -- I believe that in the bare-metal world, a more useful focus
would be to ensure that the Tuskar API service treats related group
operations (like "deploy an undercloud on these nodes") in a way that
can handle failures in a graceful and/or atomic way.


Atomicity of operations can be achieved by intoducing critical sections.
You basically have two ways of doing that, optimistic and pessimistic.
Pessimistic critical section is implemented with a locking mechanism
that prevents all other processes from entering the critical section
until it is finished.


I'm familiar with the traditional non-distributed software concept of a
mutex (or in Windows world, a critical section). But we aren't dealing
with traditional non-distributed software here. We're dealing with
highly distributed software where components involved in the
"transaction" may not be running on the same host or have much awareness
of each other at all.


Yes, that is precisely why you need to have a single point where they
can check if they are not stepping on each other's toes. If you don't,
you get race conditions and non-deterministic behavior. The only
difference with traditional, non-distributed software is that since the
components involved are communicating over a, relatively slow, network,
you have a much, much greater chance of actually having a conflict.
Scaling the whole thing to hundreds of nodes practically guarantees trouble.



Radomir, what Jay is suggesting is that it seems pretty unlikely that
two individuals would be given a directive to deploy OpenStack into a
single pool of hardware at such a scale where they will both use the
whole thing.

Worst case, if it does happen, they both run out of hardware, one
individual deletes their deployment, the other one resumes. This is the
optimistic position and it will work fine. Assuming you are driving this
all through Heat (which, AFAIK, Tuskar still uses Heat) there's even a
blueprint to support you that I'm working on:

https://blueprints.launchpad.net/heat/+spec/retry-failed-update

Even if both operators put the retry in a loop, one would actually
finish at some point.


Yes, thank you Clint. That is precisely what I was saying.


Trying to make a complex series of related but distributed actions --
like the underlying actions of the Tuskar -> Ironic API calls -- into an
atomic operation is just not a good use of programming effort, IMO.
Instead, I'm advocating that programming effort should instead be spent
coding a workflow/taskflow pipeline that can gracefully retry failed
operations and report the state of the total taskflow back to the user.


Sure, there are many ways to solve any particular synchronisation
problem. Let's say that we have one that can actually be solved by
retrying. Do you want to retry infinitely? Would you like to increase
the delays between retries exponentially? If so, where are you going to
keep the shared counters for the retries? Perhaps in tuskar-api, hmm?



I don't think a sane person would retry more than maybe once without
checking with the other operators.


Or are you just saying that we should pretend that the nondeteministic
bugs appearing due to the lack of synchronization simply don't exist?
They cannot be easily reproduced, after all. We could just close our
eyes, cover our ears, sing "lalalala" and close any bug reports with
such errors with "could not reproduce on my single-user, single-machine
development installation". I know that a lot of software companies do
exactly that, so I guess it's a valid business practice, I just want to
make sure that this is actually the tactic that we are going to take,
before commiting to an architectural decision that will make those bugs
impossible to fix.



OpenStack is non-deterministic. Deterministic systems are rigid and unable
to handle failure modes of any kind of diversity. We tend to err toward
pushing problems back to the user and giving them tools to resolve the
problem. Avoiding spurious problems is important too, no doubt. However,
what Jay has been suggesting is that the situation a pessimistic locking
system would avoid is entirely user created, and thus lower priority
than say, actually having a complete UI for deploying OpenStack.


+1. I very much agree with Jay and Clint on this matter.

Jirka

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo] [qa] oslo.config generator scrambling tempest config files

[Sean Dague]

So maybe I only just noticed it, but we moved to using the oslo.config
generator in Tempest, and it's largely succeeding at creating really
large config diffs and moving sections around all over the place.

Exhibit A: https://review.openstack.org/#/c/60434/14/etc/tempest.conf.sample

Which means it's generating a lot of noise, and I don't really
understand why. It would seem that a config generator should be order
stable. Anyone familiar enough with this to know why? And why this
doesn't seem to be an issue with other projects?

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer][Oslo] Consuming Notifications in Batches

[Gordon Sim]

On 01/02/2014 10:46 PM, Herndon, John Luke wrote:



On 1/2/14, 11:36 AM, "Gordon Sim"  wrote:


On 12/20/2013 09:26 PM, Herndon, John Luke wrote:


On Dec 20, 2013, at 12:13 PM, Gordon Sim  wrote:


On 12/20/2013 05:27 PM, Herndon, John Luke wrote:


Other protocols may support bulk consumption. My one concern with
this approach is error handling. Currently the executors treat
each notification individually. So let¹s say the broker hands
100 messages at a time. When client is done processing the
messages, the broker needs to know if message 25 had an error or
not. We would somehow need to communicate back to the broker
which messages failed. I think this may take some refactoring of
executors/dispatchers. What do you think?

[...]

(2) What would you want the broker to do with the failed messages?
What sort of things might fail? Is it related to the message
content itself? Or is it failures suspected to be of a temporal
nature?


There will be situations where the message can¹t be parsed, and those
messages can¹t just be thrown away. My current thought is that
ceilometer could provide some sort of mechanism for sending messages
that are invalid to an external data store (like a file, or a
different topic on the amqp server) where a living, breathing human
can look at them and try to parse out any meaningful information.


Right, in those cases simply requeueing probably is not the right thing
and you really want it dead-lettered in some way. I guess the first
question is whether that is part of the notification systems function,
or if it is done by the application itself (e.g. by storing it or
republishing it). If it is the latter you may not need any explicit
negative acknowledgement.


Exactly, I¹m thinking this is something we¹d build into ceilometer and not
oslo, since ceilometer is where the event parsing knowledge lives. From an
oslo point of view, the message would be 'acked¹.




Other errors might be ³database not available², in which case
re-queing the message is probably the right way to go.


That does mean however that the backlog of messages starts to grow on
the broker, so some scheme for dealing with this if the database outage
goes on for a bit is probably important. It also means that the messages
will keep being retried without any 'backoff' waiting for the database
to be restored which could increase the load.


This is a problem we already have :(


Agreed, it is a property of reliable (i.e. acknowledged) transfer from 
the broker, rather than batching. And of course, some degree of 
buffering here is exactly what message queues are supposed to provide. 
The point is simply to provide some way of configuring things so that 
this can be bounded, or prevented from taking down the entire broker. 
(And perhaps some way of altering the unfortunate someone!)



https://github.com/openstack/ceilometer/blob/master/ceilometer/notification
.py#L156-L158
Since notifications cannot be lost, overflow needs to be detected and the
messages need to be saved. I¹m thinking the database being down is a rare
occurrence that will be worthy of waking someone up in the middle of the
night. One possible solution: flip the collector into an emergency mode
and save notifications to disc until the issue is resolved. Once the db is
up and running, the collector inserts all of these saved messages (as one
big batch!). Thoughts?

I¹m not sure I understand what you are saying about retrying without a
backoff. Can you explain?


I mean that if the messages are explicitly requeued and the original 
subscription is still active, they will be immediately redelivered and 
will thus keep cycling from broker to client, back to broker, back to 
client etc etc until the database is available again.


Pulling messages off continually like this without actually being able 
to dequeue them may reduce the brokers effectiveness at e.g. paging out, 
and in any event involves some unnecessary load on top of the expanding 
queue.


It might be better, just as an example, to abort the connection to the 
broker (implicitly requeueing all unacked messages), and only reconnect 
when the database becomes available (and that can be tried after 1 
second, then 2, then 4 etc up to some maximum retry interval).


Or another alternative would be to leave the connection to the broker, 
but by not requeing or acking ensure that once the prefetch has been 
reached, no further messages will be delivered. Then locally, on the 
client, retry the processing for the prefetched messages until the 
database is back again.


The basic point I'm trying to make is that it seems to me there is 
little value in simply handing the messages back to the broker for 
immediate redelivery back to the client. It delays the retry certainly, 
but at unnecessary expense.


More generally I wonder whether an explicit negative acknowledgement is 
actually needed in the notify API at all. If it isn't, that may simplify 
things for batching.


_

Re: [openstack-dev] django-bootstrap-form & django 1.4 / 1.5

[Thomas Goirand]

On 01/02/2014 03:33 PM, Matthias Runge wrote:
> On 12/30/2013 08:31 AM, Thomas Goirand wrote:
>> Hi,
>>
>> Currently, in the global-requirements.txt, we have:
>>
>> Django>=1.4,<1.6
>> django-bootstrap-form
>>
>> However, django-bootstrap-form fail in both Django 1.4 and Django 1.6.
>>
>> What's the way forward? Would it be possible that someone makes a patch
>> for django-bootstrap-form, so that it could work with Django 1.4 & 1.5
>> (which would be the best way forward for me...)?
>>
> AFAIK, horizon is the only project under the OpneStack umbrella using
> django. In horizons requirements.txt, django-bootstrap-form is missing.
> 
> Djangp-bootstrap-form was added to global requirements by[1], but is
> seems it's not used at all, so I propose to remove it.
> 
> Matthias
> 
> 
> [1]
> https://github.com/openstack/requirements/commit/46a6f06326895b7293cfc0714acce25e1e3d6159

If we don't need it, then yes, please remove it ASAP.

Thomas


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Novnc switch to sockjs-client instead of websockify

[Thomas Goirand]

On 01/03/2014 01:03 AM, Ben Nemec wrote:
> On 2014-01-01 00:33, Thomas Goirand wrote:
>> Hi,
>>
>> I was wondering if it would be possible for NoVNC to switch from
>> websockify to sockjs-client, which is available here:
>>
>> https://github.com/sockjs/sockjs-client
>>
>> This has the advantage of not using flash at all (pure javascript), and
>> continuing to work on all browsers, with a much cleaner licensing.
>>
>> Thoughts welcome!
>>
>> Cheers,
>>
>> Thomas Goirand (zigo)
> 
> Sounds reasonable, but NoVNC isn't an OpenStack project so you'll need
> to bring this up with its developers: http://kanaka.github.io/noVNC/
> 
> -Ben

Well, this is just an idea I'm sending to the list. I know that the
current upstream for NoVNC isn't paid (anymore?) for his work on NoVNC,
and I will not have the time to work on this myself.

My hope was that someone in this list would catch up...

Thomas


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon] Javascript checkstyle improvement

[Maxime Vidori]

Someone said Grunt?

- Original Message -
From: "Matthias Runge" 
To: openstack-dev@lists.openstack.org
Sent: Friday, January 3, 2014 9:52:48 AM
Subject: Re: [openstack-dev] [horizon] Javascript checkstyle improvement

On 01/03/2014 09:37 AM, Radomir Dopieralski wrote:

> [snip]
> 
> This is actually not a problem at all, because the way jshint works now,
> we have to explicitly list the files to be checked against those
> rules. That means, that we can only check our own code, and not the
> included libraries. Of course, the reviewers have to pay attention to
> make sure that all the non-library code is added to the list, but we
> don't really get new files that often.
> 
Yeah, jshint needs to be switched on explicitly for each file. That
tends to be forgotten on new files, which is not ideal either.

Matthias

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] MySQLdb situation with the gate

[Flavio Percoco]

On 02/01/14 17:41 -0600, Monty Taylor wrote:



On 01/02/2014 12:48 PM, Monty Taylor wrote:

Hey all!

This morning, we experienced our first systemic breakage of the gate of
2014! There are a few reasons for this, which are related to pip and
virtualenv making new releases. The new releases do WONDERFUL things and
are more secure, but unfortunately, we've hit a few pain points which we
are working through.

The most visible one that's going to come across people's plate has to
do with mysql-python. The upstream package still hasn't updated itself
from attempting to explicitly use distribute, and it cannot be installed
by pip 1.5. (this has nothing to do with OpenStack - mysql-python is
simply uninstallable with pip 1.5 at the moment)

Since we kinda use the heck out of mysql-python, and we're also pretty
heavy users of pip, we have to address the situation.

First of all, we made a pull request to upstream:

 https://github.com/farcepest/MySQLdb1/pull/44

That fixes the issue. In a perfect world, this will get merged and 1.2.5
will get released in the next 30 minutes.


Holla! A perfect world happened and Andy Dustman has merged the PR and
cut a 1.2.5 release.


;( Tears of happiness!!!

--
@flaper87
Flavio Percoco


pgpyC0kCl3QHz.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][qa] Parallel testing update

[Salvatore Orlando]

I have already a patch under review for the quota test, for which I adopted
the shortest-diff approach.
As regards Robert's suggestion, the problem we have there is that the test
uses a dedicated tenant, but it does not take into account the possibility
that at some point the dhcp agent will create a port too for that tenant.

In theory I tend to agree with Miguel; but I'm not sure what would be the
consensus on removing a scenario test. I think we either decide to merge
this shortest-diff patch [1] once the comments are addressed, or re-design
the tests, which might take some more time.

Salvatore

PS: shortest-diff is, as you might have already understood, an euphemism
for 'hack'


[1] https://review.openstack.org/#/c/64217/



On 2 January 2014 22:39, Robert Collins  wrote:

> Another way to tackle it would be to create a dedicated tenant for
> those tests, then the quota won't interact with anything else.
>
> On 3 January 2014 10:35, Miguel Angel Ajo Pelayo 
> wrote:
> > Hi Salvatore!,
> >
> >Good work on this.
> >
> >About the quota limit tests, I believe they may be unit-tested,
> > instead of functionally tested.
> >
> >When running those tests in parallel with any other tests that rely
> > on having ports, networks or subnets available into quota, they have
> > high chances of making those other tests fail.
> >
> > Cheers,
> > Miguel Ángel Ajo
> >
> >
> >
> > - Original Message -
> >> From: "Kyle Mestery" 
> >> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> >> Sent: Thursday, January 2, 2014 7:53:05 PM
> >> Subject: Re: [openstack-dev] [Neutron][qa] Parallel testing update
> >>
> >> Thanks for the updates here Salvatore, and for continuing to push on
> >> this! This is all great work!
> >>
> >> On Jan 2, 2014, at 6:57 AM, Salvatore Orlando 
> wrote:
> >> >
> >> > Hi again,
> >> >
> >> > I've now run the experimental job a good deal of times, and I've
> filed bugs
> >> > for all the issues which came out.
> >> > Most of them occurred no more than once among all test execution (I
> think
> >> > about 30).
> >> >
> >> > They're all tagged with neutron-parallel [1]. for ease of tracking,
> I've
> >> > associated all the bug reports with neutron, but some are probably
> more
> >> > tempest or nova issues.
> >> >
> >> > Salvatore
> >> >
> >> > [1]
> https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel
> >> >
> >> >
> >> > On 27 December 2013 11:09, Salvatore Orlando 
> wrote:
> >> > Hi,
> >> >
> >> > We now have several patches under review which improve a lot how
> neutron
> >> > handles parallel testing.
> >> > In a nutshell, these patches try to ensure the ovs agent processes
> new,
> >> > removed, and updated interfaces as soon as possible,
> >> >
> >> > These patches are:
> >> > https://review.openstack.org/#/c/61105/
> >> > https://review.openstack.org/#/c/61964/
> >> > https://review.openstack.org/#/c/63100/
> >> > https://review.openstack.org/#/c/63558/
> >> >
> >> > There is still room for improvement. For instance the calls from the
> agent
> >> > into the plugins might be consistently reduced.
> >> > However, even if the above patches shrink a lot the time required for
> >> > processing a device, we are still hitting a hard limit with the
> execution
> >> > ovs commands for setting local vlan tags and clearing flows (or
> adding the
> >> > flow rule for dropping all the traffic).
> >> > In some instances this commands slow down a lot, requiring almost 10
> >> > seconds to complete. This adds a delay in interface processing which
> in
> >> > some cases leads to the hideous SSH timeout error (the same we see
> with
> >> > bug 1253896 in normal testing).
> >> > It is also worth noting that when this happens sysstat reveal CPU
> usage is
> >> > very close to 100%
> >> >
> >> > From the neutron side there is little we can do. Introducing parallel
> >> > processing for interface, as we do for the l3 agent, is not actually a
> >> > solution, since ovs-vswitchd v1.4.x, the one executed on gate tests,
> is
> >> > not multithreaded. If you think the situation might be improved by
> >> > changing the logic for handling local vlan tags and putting ports on
> the
> >> > dead vlan, I would be happy to talk about that.
> >> > On my local machines I've seen a dramatic improvement in processing
> times
> >> > by installing ovs 2.0.0, which has a multi-threaded vswitchd. Is this
> >> > something we might consider for gate tests? Also, in order to reduce
> CPU
> >> > usage on the gate (and making tests a bit faster), there is a tempest
> >> > patch which stops creating and wiring neutron routers when they're not
> >> > needed: https://review.openstack.org/#/c/62962/
> >> >
> >> > Even in my local setup which succeeds about 85% of times, I'm still
> seeing
> >> > some occurrences of the issue described in [1], which at the end of
> the
> >> > day seems a dnsmasq issue.
> >> >
> >> > Beyond the 'big' structural problem 

Re: [openstack-dev] [horizon] Javascript checkstyle improvement

[Matthias Runge]

On 01/03/2014 09:37 AM, Radomir Dopieralski wrote:

> [snip]
> 
> This is actually not a problem at all, because the way jshint works now,
> we have to explicitly list the files to be checked against those
> rules. That means, that we can only check our own code, and not the
> included libraries. Of course, the reviewers have to pay attention to
> make sure that all the non-library code is added to the list, but we
> don't really get new files that often.
> 
Yeah, jshint needs to be switched on explicitly for each file. That
tends to be forgotten on new files, which is not ideal either.

Matthias

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] Common SSH

[Flavio Percoco]

On 02/01/14 11:17 -0500, Doug Hellmann wrote:




On Mon, Dec 30, 2013 at 5:23 AM, Flavio Percoco  wrote:

   On 26/12/13 20:05 +0200, Sergey Skripnick wrote:


   Hi all,

   I'm surprised there is no common ssh library in oslo so I filed this
   blueprint[0]. I would be happy to address any comments/suggestions.


   [0] https://blueprints.launchpad.net/oslo/+spec/common-ssh-client



   If you think the BP is ready to be reviewed - which I bet you do since
   you already submitted a patch - please, set a target milestone and
   series. That'll make it pop-up into the BPs review process.


I left a comment on the blueprint, but I'll post to the list for a broader
audience.

I'd like to explore whether the paramiko team will accept this code (or
something like it). This seems like a perfect opportunity for us to contribute
upstream.


+1

The patch is not big and the code seems simple and reasonable enough
to live within paramiko.

Cheers,
FF


--
@flaper87
Flavio Percoco


pgpO0EaRIvpKu.pgp
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [horizon] Javascript checkstyle improvement

[Radomir Dopieralski]

On 03/01/14 08:17, Matthias Runge wrote:
> On 12/27/2013 03:52 PM, Maxime Vidori wrote:

>> I send this mail to talk about Javascript coding style improvement,
>> like python has pep8, it could be interesting to have some rules for
>> javascript too. JSHint provides some rules to perform this and I think
>> it could be a great idea to discuss about which rules could be integrated
>> into Horizon.

[snip]

> We're bundling foreign code (which is bad in general); now we need to
> change/format that code too, to match our style conventions? That would
> even generate a fork, like here[1], where the changes were just cosmetics.

[snip]

This is actually not a problem at all, because the way jshint works now,
we have to explicitly list the files to be checked against those
rules. That means, that we can only check our own code, and not the
included libraries. Of course, the reviewers have to pay attention to
make sure that all the non-library code is added to the list, but we
don't really get new files that often.

-- 
Radomir Dopieralski

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev