Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Ben Nemec
On 11/30/2015 02:15 PM, Sean Dague wrote:
> On 11/30/2015 03:01 PM, Robert Collins wrote:
>> On 1 December 2015 at 08:37, Ben Nemec  wrote:
>>> On 11/30/2015 12:42 PM, Joshua Harlow wrote:
 Hi all,

 I just wanted to bring up an issue, possible solution and get feedback
 on it from folks because it seems to be an on-going problem that shows
 up not when an application is initially deployed but as on-going
 operation and running of that application proceeds (ie after running for
 a period of time).

 The jist of the problem is the following:

 A <> has a need to ensure that no
 application on the same machine can manipulate a given resource on that
 same machine, so it uses the lock file pattern (acquire a *local* lock
 file for that resource, manipulate that resource, release that lock
 file) to do actions on that resource in a safe manner (note this does
 not ensure safety outside of that machine, lock files are *not*
 distributed locks).

 The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
 external=False, lock_path=None, semaphores=None, delay=0.01)

 or via its underlying library (that I extracted from oslo.concurrency
 and have improved to add more usefulness) @
 http://fasteners.readthedocs.org/

 The issue though for <> is that each of
 these projects now typically has a large amount of lock files that exist
 or have existed and no easy way to determine when those lock files can
 be deleted (afaik no? periodic task exists in said projects to clean up
 lock files, or to delete them when they are no longer in use...) so what
 happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
 appear and there is no a simple solution to clean lock files up (since
 oslo.concurrency is really not the right layer to know when a lock can
 or can not be deleted, only the application knows that...)

 So then we get a few creative solutions like the following:

 - https://review.openstack.org/#/c/241663/
 - https://review.openstack.org/#/c/239678/
 - (and others?)

 So I wanted to ask the question, how are people involved in <>>> favorite openstack project>> cleaning up these files (are they at all?)

 Another idea that I have been proposing also is to use offset locks.

 This would allow for not creating X lock files, but create a *single*
 lock file per project and use offsets into it as the way to lock. For
 example nova could/would create a 1MB (or larger/smaller) *empty* file
 for locks, that would allow for 1,048,576 locks to be used at the same
 time, which honestly should be way more than enough, and then there
 would not need to be any lock cleanup at all... Is there any reason this
 wasn't initially done back way when this lock file code was created?
 (https://github.com/harlowja/fasteners/pull/10 adds this functionality
 to the underlying library if people want to look it over)
>>>
>>> I think the main reason was that even with a million locks available,
>>> you'd have to find a way to hash the lock names to offsets in the file,
>>> and a million isn't a very large collision space for that.  Having two
>>> differently named locks that hashed to the same offset would lead to
>>> incredibly confusing bugs.
>>>
>>> We could switch to requiring the projects to provide the offsets instead
>>> of hashing a string value, but that's just pushing the collision problem
>>> off onto every project that uses us.
>>>
>>> So that's the problem as I understand it, but where does that leave us
>>> for solutions?  First, there's
>>> https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
>>> which allows consumers to delete lock files when they're done with them.
>>>  Of course, in that case the onus is on the caller to make sure the lock
>>> couldn't possibly be in use anymore.
>>>
>>> Second, is this actually a problem?  Modern filesystems have absurdly
>>> large limits on the number of files in a directory, so it's highly
>>> unlikely we would ever exhaust that, and we're creating all zero byte
>>> files so there shouldn't be a significant space impact either.  In the
>>> past I believe our recommendation has been to simply create a cleanup
>>> job that runs on boot, before any of the OpenStack services start, that
>>> deletes all of the lock files.  At that point you know it's safe to
>>> delete them, and it prevents your lock file directory from growing forever.
>>
>> Not that high - ext3 (still the default for nova ephemeral
>> partitions!) has a limit of 64k in one directory.
>>
>> That said, I don't disagree - my thinkis is that we should advise
>> putting such files on a tmpfs.
> 
> So, I think the issue really is that the named 

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Joshua Harlow

Ben Nemec wrote:

On 11/30/2015 02:15 PM, Sean Dague wrote:

On 11/30/2015 03:01 PM, Robert Collins wrote:

On 1 December 2015 at 08:37, Ben Nemec  wrote:

On 11/30/2015 12:42 PM, Joshua Harlow wrote:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<>  has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<>  is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<>  cleaning up these files (are they at all?)

Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)

I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that.  Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions?  First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
which allows consumers to delete lock files when they're done with them.
  Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.

Second, is this actually a problem?  Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either.  In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files.  At that point you know it's safe to
delete them, and it prevents your lock file directory from growing forever.

Not that high - ext3 (still the default for nova ephemeral
partitions!) has a limit of 64k in one directory.

That said, I don't disagree - my thinkis is that we should advise
putting such files on a tmpfs.

So, I think the issue really is that the named external locks were
originally thought to be handling some pretty sensitive critical
sections. Both cinder / nova have less than 20 such named locks.

Cinder uses a parametrized version for all volume operations -
https://github.com/openstack/cinder/blob/7fb767f2d652f070a20fd70d92585d61e56f3a50/cinder/volume/manager.py#L143


Nova also does something similar in image cache

Re: [openstack-dev] [TripleO/heat] openstack debug command

2015-11-30 Thread Steven Hardy
On Tue, Dec 01, 2015 at 08:47:20AM +1300, Steve Baker wrote:
> On 30/11/15 23:21, Steven Hardy wrote:
> >On Mon, Nov 30, 2015 at 10:03:29AM +0100, Lennart Regebro wrote:
> >>I'm tasked to implement a command that shows error messages when a
> >>deployment has failed. I have a vague memory of having seen scripts
> >>that do something like this, if that exists, can somebody point me in
> >>teh right direction?
> >I wrote a super simple script and put it in a blog post a while back:
> >
> >http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-3-cluster.html
> >
> >All it does is find the failed SoftwareDeployment resources, then do heat
> >deployment-show on the resource, so you can see the stderr associated with
> >the failure.
> >
> >Having tripleoclient do that by default would be useful.
> >
> >>Any opinions on what that should do, specifically? Traverse failed
> >>resources to find error messages, I assume. Anything else?
> >Yeah, but I think for this to be useful, we need to go a bit deeper than
> >just showing the resource error - there are a number of typical failure
> >modes, and I end up repeating the same steps to debug every time.
> >
> >1. SoftwareDeployment failed (mentioned above).  Every time, you need to
> >see the name of the SoftwareDeployment which failed, figure out if it
> >failed on one or all of the servers, then look at the stderr for clues.
> >
> >2. A server failed to build (OS::Nova::Server resource is FAILED), here we
> >need to check both nova and ironic, looking first to see if ironic has the
> >node(s) in the wrong state for scheduling (e.g nova gave us a no valid
> >host error), and then if they are OK in ironic, do nova show on the failed
> >host to see the reason nova gives us for it failing to go ACTIVE.
> >
> >3. A stack timeout happened.  IIRC when this happens, we currently fail
> >with an obscure keystone related backtrace due to the token expiring.  We
> >should instead catch this error and show the heat stack status_reason,
> >which should say clearly the stack timed out.
> >
> >If we could just make these three cases really clear and easy to debug, I
> >think things would be much better (IME the above are a high proportion of
> >all failures), but I'm sure folks can come up with other ideas to add to
> >the list.
> >
> I'm actually drafting a spec which includes a command which does this. I
> hope to submit it soon, but here is the current state of that command's
> description:
> 
> Diagnosing resources in a FAILED state
> --
> 
> One command will be implemented:
> - openstack overcloud failed list
> 
> This will print a yaml tree showing the hierarchy of nested stacks until it
> gets to the actual failed resource, then it will show information regarding
> the
> failure. For most resource types this information will be the status_reason,
> but for software-deployment resources the deploy_stdout, deploy_stderr and
> deploy_status code will be printed.
> 
> In addition to this stand-alone command, this output will also be printed
> when
> an ``openstack overcloud deploy`` or ``openstack overcloud update`` command
> results in a stack in a FAILED state.

This sounds great!

Another piece of low-hanging-fruit in the meantime is we should actually
print the stack_status_reason on failure:

https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L280

The DeploymentError raised could include the stack_status_reason vs the
unqualified "Heat Stack create failed".

I guess your event listing partially overlaps with this, as you can now
derive the stack_status_reason from the last event, but it's still be good
to loudly output it so folks can see more quickly when things such as
timeouts happen that are clearly displayed in the top-level stack status.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Steven Hardy
On Mon, Nov 30, 2015 at 03:19:18PM +, Derek Higgins wrote:
> Hi All,
> 
> A few months tripleo switch from its devtest based CI to one that was
> based on instack. Before doing this we anticipated disruption in the ci jobs
> and removed them from non tripleo projects.
> 
> We'd like to investigate adding it back to heat and ironic as these are
> the two projects where we find our ci provides the most value. But we can
> only do this if the results from the job are treated as voting.
> 
> In the past most of the non tripleo projects tended to ignore the
> results from the tripleo job as it wasn't unusual for the job to broken for
> days at a time. The thing is, ignoring the results of the job is the reason
> (the majority of the time) it was broken in the first place.
> To decrease the number of breakages we are now no longer running master
> code for everything (for the non tripleo projects we bump the versions we
> use periodically if they are working). I believe with this model the CI jobs
> we run have become a lot more reliable, there are still breakages but far
> less frequently.
> 
> What I proposing is we add at least one of our tripleo jobs back to both
> heat and ironic (and other projects associated with them e.g. clients,
> ironicinspector etc..), tripleo will switch to running latest master of
> those repositories and the cores approving on those projects should wait for
> a passing CI jobs before hitting approve. So how do people feel about doing
> this? can we give it a go? A couple of people have already expressed an
> interest in doing this but I'd like to make sure were all in agreement
> before switching it on.

+1 - TripleO has quite frequently encountered heat related bugs in the
past, and it'd be good to catch those earlier if at all possible.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Jeremy Stanley
On 2015-11-30 19:54:33 + (+), Paul Michali wrote:
> Check out https://freenode.net/irc_servers.shtml which lists the servers. I
> was using irc.freenode.net. Switched to weber.freenode.net and able to
> connect.
> 
> (now everyone will hop on that one and I'll have to pick another :)

Thanks--I (perhaps incorrectly?) assumed that only those they
maintain in the chat.freenode.net DNS round-robin are considered
active.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Steve Baker

On 01/12/15 04:19, Derek Higgins wrote:

Hi All,

A few months tripleo switch from its devtest based CI to one that 
was based on instack. Before doing this we anticipated disruption in 
the ci jobs and removed them from non tripleo projects.


We'd like to investigate adding it back to heat and ironic as 
these are the two projects where we find our ci provides the most 
value. But we can only do this if the results from the job are treated 
as voting.


In the past most of the non tripleo projects tended to ignore the 
results from the tripleo job as it wasn't unusual for the job to 
broken for days at a time. The thing is, ignoring the results of the 
job is the reason (the majority of the time) it was broken in the 
first place.
To decrease the number of breakages we are now no longer running 
master code for everything (for the non tripleo projects we bump the 
versions we use periodically if they are working). I believe with this 
model the CI jobs we run have become a lot more reliable, there are 
still breakages but far less frequently.


What I proposing is we add at least one of our tripleo jobs back to 
both heat and ironic (and other projects associated with them e.g. 
clients, ironicinspector etc..), tripleo will switch to running latest 
master of those repositories and the cores approving on those projects 
should wait for a passing CI jobs before hitting approve. So how do 
people feel about doing this? can we give it a go? A couple of people 
have already expressed an interest in doing this but I'd like to make 
sure were all in agreement before switching it on.


+1 for heat from me. It sounds like the job won't be voting, but heat 
cores should be strongly encouraged to treat it as such.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [TripleO/heat] openstack debug command

2015-11-30 Thread Lennart Regebro
I'm tasked to implement a command that shows error messages when a
deployment has failed. I have a vague memory of having seen scripts
that do something like this, if that exists, can somebody point me in
teh right direction?

Any opinions on what that should do, specifically? Traverse failed
resources to find error messages, I assume. Anything else?

//Lennart

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [release] using reno for libraries

2015-11-30 Thread Dmitry Tantsur

On 11/28/2015 02:48 PM, Doug Hellmann wrote:

Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:

Liaisons,

We're making good progress on adding reno to service projects as
we head to the Mitaka-1 milestone. Thank you!

We also need to add reno to all of the other deliverables with
changes that might affect deployers. That means clients and other
libraries, SDKs, etc. with configuration options or where releases
can change deployment behavior in some way. Now that most teams
have been through this conversion once, it should be easy to replicate
for the other repositories in a similar way.

Libraries have 2 audiences for release notes: developers consuming
the library and deployers pushing out new versions of the libraries.
To separate the notes for the two audiences, and avoid doing manually
something that we have been doing automatically, we can use reno
just for deployer release notes (changes in support for options,
drivers, etc.). That means the library repositories that need reno
should have it configured just like for the service projects, with
the separate jobs and a publishing location different from their
existing developer documentation. The developer docs can continue
to include notes for the developer audience.


I've had a couple of questions about this split for release notes. The
intent is for developer-focused notes to continue to come from commit
messages and in-tree documentation, while using reno for new and
additional deployer-focused communication. Most commits to libraries
won't need reno release notes.


This looks like unnecessary overcomplication. Why not use such a 
convenient tool for both kinds of release notes instead of having us 
invent and maintain one more place to put release notes, now for 
developers? It's already not so easy to explain reno to newcomers, this 
idea makes it even harder...




Doug



After we start using reno for libraries, the release announcement
email tool will be updated to use those same notes to build the
message in addition to looking at the git change log. This will be
a big step toward unifying the release process for services and
libraries, and will allow us to make progress on completing the
automation work we have planned for this cycle.

It's not necessary to add reno to the liberty branch for library
projects, since we tend to backport far fewer changes to libraries.
If you maintain a library that does see a lot of backports, by all
means go ahead and add reno, but it's not a requirement. If you do
set up multiple branches, make sure you have one page that uses the
release-notes directive without specifing a branch, as in the
oslo.config example, to build notes for the "current" branch to get
releases from master and to serve as a test for rendering notes
added to stable branches.

Thanks,
Doug



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][vpnaas] "SKIPPED: Neutron support is required" while running tox

2015-11-30 Thread Ihar Hrachyshka

bharath  wrote:


Hi,

when running tox "sudo -u stack -H tox -e api  
neutron.tests.api.test_vpnaas_extensions"
test cases are failing with Error " setUpClass  
(neutron.tests.api.test_vpnaas_extensions.VPNaaSTestJSON) ... SKIPPED:  
Neutron support is required"


I even tried setting tempest_config_dir as below, still i am hitting the  
issue.


[testenv:api]
basepython = python2.7
passenv = {[testenv]passenv} TEMPEST_CONFIG_DIR
setenv = {[testenv]setenv}
 OS_TEST_PATH=./neutron/tests/api
 TEMPEST_CONFIG_DIR={env:TEMPEST_CONFIG_DIR:/opt/stack/tempest/etc}
 OS_TEST_API_WITH_REST=1


Can someone help me out



Does your /opt/stack/tempest/etc have tempest configured to run tests for  
neutron?


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Magnum] Liberty RPMs for RDO

2015-11-30 Thread Mathieu Velten
Hi,

Let me first introduce myself : I am currently working at CERN to help
evaluate and deploy Magnum.

In this regard Ricardo recently sends an email regarding Puppet
modules, this one is about RPMs of Magnum for CentOS with RDO.

You can find here a repository containing the source and binary RPMs
for magnum and python-magnumclient.
http://linuxsoft.cern.ch/internal/repos/magnum7-testing/

The version 1.0.0.0b2.dev4 is the Magnum Liberty release and the
1.1.0.0-5 version is the Mitaka M1 release using Liberty dependencies
(one client commit regarding keystone auth and one server commit
regarding oslo.config have been reverted).

Let me know how I can contribute the spec files to somewhere more
suitable.

Regards,

Mathieu Velten
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] IRC Bot issues

2015-11-30 Thread Joshua Hesketh
Hi all,

Freenode are currently experiencing a severe DDoS attack that are having an
effect on our bots. As such the meetbot, irc logging and gerrit watcher are
interminably available.

We expect the bots to resume their normal function once Freenode has
recovered. For now, meetings may have to be postponed or minuted by hand.

Cheers,
Josh
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread hao wang
Hi, Duncan

2015-11-30 15:54 GMT+08:00 Duncan Thomas :
> Hi WangHao
>
> This was quite thoroughly discussed during the early discussions on
> replication. The general statement was 'not yet'. Getting any kind of
> workable replication API has proven to be very, very difficult to get right
> - we won't know for another full cycle whether we've actually gotten it
> somewhere near right, as operators start to deploy it. Piling more feature
> in the replication API before a) it has been used by operators and b)
> storage vendors have implemented what we already has would IMO be a mistake.

I agree with you, in my mind, using replication what we have is first
thing we should done,
improve it much better is second thing, and then we will add another
new features
one by one stably.

> None of this means that more DR interfaces don't belong in cinder, just that
> getting them right, getting them universal and getting them useful is quite
> a hard problem, and not one we should be in a rush to solve. Particularly as
> DR and replication is still a niche area of cinder, and we still have major
> issues in our basic functionality.

Yes, this will convince me about DR in Cinder, very clearly, thanks.
>
> On 30 November 2015 at 03:45, hao wang  wrote:
>>
>> Sean and Michal,
>>
>> In fact, there is a reason that I ask this question. Recently I have a
>> confusion about if cinder should provide the ability of Disaster
>> Recovery to storage resources, like volume. I mean we have volume
>> replication v1, but for DR, specially DR between two independent
>> OpenStack sites(production and DR site), I feel we still need more
>> features to support it, for example consistency group for replication,
>> etc. I'm not sure if those features belong in Cinder or some new
>> project for DR.
>>
>> BR
>> WangHao
>>
>> 2015-11-30 3:02 GMT+08:00 Sean McGinnis :
>> > On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
>> >> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
>> >> > Hi guys,
>> >> >
>> >> > I notice nova have a clarification of project scope:
>> >> > http://docs.openstack.org/developer/nova/project_scope.html
>> >> >
>> >> > I want to find cinder's, but failed,  do you know where to find it?
>> >> >
>> >> > It's important to let developers know what feature should be
>> >> > introduced into cinder and what shouldn't.
>> >> >
>> >> > BR
>> >> > Wang Hao
>> >>
>> >> I believe Nova team needed to formalize the scop to have an explanation
>> >> for all the "this doesn't belong in Nova" comments on feature requests.
>> >> Does Cinder suffer from similar problems? From my perspective it's not
>> >> critically needed.
>> >
>> > I agree. I haven't seen a need for something like that with Cinder. Wang
>> > Hao, is there a reason you feel you need that?
>> >
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> --
> Duncan Thomas
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [nova][RFC] delete instance device

2015-11-30 Thread 少合冯
Hi all,

I'd like to talk about the delete instance device of nova.

Here is the libvirt doc string to describe it underly function
detachDeviceFlags.
http://paste.openstack.org/show/480330/

It says:

detaching a device from a running domain may be asynchronous.

*and it suggests:*

To check whether the device was successfully removed, either recheck domain

configuration using virDomainGetXMLDesc() or add handler for

VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED


Also Daniel elaborated it, and gave us more some scenarios about it.

it is not guaranteed to succeed. What happens is that the hypervisor
injects an ACPI request to unplug the device. The guest OS must co-operate
by releasing the device, before the hypervisor will complete the action of
physically removing it. So you require a guest OS that supports ACPI unplug
of course, and if the guest is crashed or being malicious there is no
guarantee the unplug will succeed. Libvirt will wait a short while for
success, but you must monitor for libvirt events to see if/when it finally
completes. This delayed release has implications for when Nova can mark the
PCI device as unused and available for other guests to assign.


Now I have checked the code, both detach volume or detach interface call
   5   1220  nova/virt/libvirt/driver.py <>
 guest . detach_device ( conf , persistent = True , live = live
)
   6   1280  nova/virt/libvirt/driver.py <>
 guest . detach_device ( cfg , persistent = True , live = live )
   7   3016  nova/virt/libvirt/driver.py <<_detach_pci_devices>>
 guest . detach_device ( self . _get_guest_pci_device ( dev ) ,
live = True )
   8   3105  nova/virt/libvirt/driver.py <<_detach_sriov_ports>>
 guest . detach_device ( cfg , live = True )

And for detach_interface in nova/compute/manager.py:

@wrap_exception()
@wrap_instance_fault
def detach_interface(self, context, instance, port_id):
"""Detach an network adapter from an instance."""
network_info = instance.info_cache.network_info
condemned = None
for vif in network_info:
if vif['id'] == port_id:
condemned = vif
break
if condemned is None:
raise exception.PortNotFound(_("Port %s is not "
   "attached") % port_id)
try:
self.driver.detach_interface(instance, condemned)
except exception.NovaException as ex:
LOG.warning(_LW("Detach interface failed, port_id=%(port_id)s,"
" reason: %(msg)s"),
{'port_id': port_id, 'msg': ex}, instance=instance)
raise exception.InterfaceDetachFailed(instance_uuid=instance.uuid)
else:
try:
self.network_api.deallocate_port_for_instance(
context, instance, port_id)
except Exception as ex:
with excutils.save_and_reraise_exception():
# Since this is a cast operation, log the failure for
# triage.
LOG.warning(_LW('Failed to deallocate port %(port_id)s '
'for instance. Error: %(error)s'),
{'port_id': port_id, 'error': ex},
instance=instance)



It just detach_interface, no double check the device is detached finally.

Now I will support the detach SRIOV code.
https://review.openstack.org/#/c/139910/
I'm not sure should I need to double check the device is finally detached.

If yes.

What should I support?

3 options:
1. Just ignored it. key the nova code.

2. sync check.
as the libvirt suggests: use virDomainGetXMLDesc()

def detach_interface(self, context, instance, port_id):

   self.driver.detach_interface(instance, condemned)

   # just *pseudo-code*

*   for i in range(1, 51):*

  if not(virDomainGetXMLDesc()):

   sleep(1)

   else if i == 51:

   raise exception

   else:

   break

   self.network_api.deallocate_port_for_instance(
   context, instance, port_id)



3. async notification.
 as the libvirt suggests:

add event handler for VIR_DOMAIN_EVENT_ID_DEVICE_REMOVED.


call network_api.deallocate_port_for_instance in a backend task.
The backend receives the event result from event handler by AMPQ and filter
the
device is the the expected interface device, not the volume device.

Then backend call network_api.deallocate_port_for_instance to deallocate
the port.


I have not check the volume detach, not sure it has the same issues.

Beside this issue:
But from the libvirt Doc string,

hypervisors may prevent this operation if there is a currentblock copy
operation on the device being detached;
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] 答复: [congress]role in Congress

2015-11-30 Thread Masahito MUROI

Hi Yali,

I'm not sure which your concern is about UI by Horizon or for a policy 
in Congress.


On 2015/11/30 16:27, zhangyali (D) wrote:

Hi Tim,

In the implementation of Congress UI, I have realized that we need to
add role assignment. In many systems and cases, RBAC (Role-Based Access
Control) is an vital function which are beneficial to the division of work.

I think we should have “admin” and “tenant” user at least.  The admin
user could define predicates or complicated relations used in the
violations, and tenant user could just use the predicates defined by
admin user.

A typical example is that, the owner of network connected to VM should
be in the same group with the owner of this VM. In this example,
same_group is a predicates or complicated relations. It should be
I think this depends on how admin writes policy and what service works 
as Congress datasource. If admin want to adopt the policy to specific 
tenant, user or group, admin writes an additional policy to affect them 
by the policy.




defined by admin user. And this predicate could be used by any tenant
user. In this way, some typical or common predicates could be defined in
a individual admin board, and other tenant user could choose which one
to use.
On the other hand, this mentions about which user has permission to edit 
and to see a policy by Horizon.


best regard,
Masahito



Yali

*发件人:*Tim Hinrichs [mailto:t...@styra.com]
*发送时间:*2015年11月30日10:37
*收件人:*zhangyali (D); OpenStack Development Mailing List (not for
usage questions)
*主题:*Re: [openstack-dev][congress]role in Congress

Hi Yali,

We didn't discuss role assignment for service users in Tokyo.  Could you
explain a bit more what you need?

Tim

On Sun, Nov 29, 2015 at 7:33 PM zhangyali (D) > wrote:

Hi Tim and All,

I remember there is a topic named “role assignment for service
users”in the Tokyo Summit. Since I have not heard any message of
this topic. Does anyone could contribute some information for me? I
think it is vital for my design of Congress UI in horizon. Thanks a lot!

Best Regards,

Yali



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




--
室井 雅仁(Masahito MUROI)
Software Innovation Center, NTT
Tel: +81-422-59-4539,FAX: +81-422-59-2699



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [magnum] Autoscaling both clusters and containers

2015-11-30 Thread Peng Zhao

Ryan,
That's where Hyper could help. This blog talks about wasted capacity issue and
the solution:
http://thenewstack.io/hypernetes-brings-multi-tenancy-microservices/
Best Peng
- Hyper - Make VM run like 
Container


On Wed, Nov 18, 2015 at 6:03 AM, Ryan Rossiter < rlros...@linux.vnet.ibm.com > 
wrote:
Hi all,

I was having a discussion with a teammate with respect to container scaling. He
likes the aspect of nova-docker that allows you to scale (essentially)
infinitely almost instantly, assuming you are using a large pool of compute
hosts. In the case of Magnum, if I'm a container user, I don't want to be paying
for a ton of vms that just sit idle, but I also want to have enough vms to
handle my scale when I infrequently need it. But above all, when I need scale, I
don't want to suddenly have to go boot vms and wait for them to start up when I
really need it.

I saw [1] which discusses container scaling, but I'm thinking we can take this
one step further. If I don't want to pay for a lot of vms when I'm not using
them, could I set up an autoscale policy that allows my cluster to expand when
my container concentration gets too high on my existing cluster? It's kind of a
case of nested autoscaling. The containers are scaled based on request demand,
and the cluster vms are scaled based on container count.

I'm unsure of the details of Senlin, but at least looking at Heat autoscaling
[2], this would not be very hard to add to the Magnum templates, and we would
forward those on through the bay API. (I figure we would do this through the
bay, not baymodel, because I can see similar clusters that would want to be
scaled differently).

Let me know if I'm totally crazy or if this is a good idea (or if you guys have
already talked about this before). I would be interested in your feedback.

[1] http://lists.openstack.org/pip ermail/openstack-dev/2015-Nove 
mber/078628.html
[2] https://wiki.openstack.org/wik i/Heat/AutoScaling#AutoScaling _API

--
Thanks,

Ryan Rossiter (rlrossit)


__ __ __
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.op enstack.org?subject:unsubscrib e
http://lists.openstack.org/cgi -bin/mailman/listinfo/openstac k-dev__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO/heat] openstack debug command

2015-11-30 Thread Steven Hardy
On Mon, Nov 30, 2015 at 10:03:29AM +0100, Lennart Regebro wrote:
> I'm tasked to implement a command that shows error messages when a
> deployment has failed. I have a vague memory of having seen scripts
> that do something like this, if that exists, can somebody point me in
> teh right direction?

I wrote a super simple script and put it in a blog post a while back:

http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-3-cluster.html

All it does is find the failed SoftwareDeployment resources, then do heat
deployment-show on the resource, so you can see the stderr associated with
the failure.

Having tripleoclient do that by default would be useful.

> Any opinions on what that should do, specifically? Traverse failed
> resources to find error messages, I assume. Anything else?

Yeah, but I think for this to be useful, we need to go a bit deeper than
just showing the resource error - there are a number of typical failure
modes, and I end up repeating the same steps to debug every time.

1. SoftwareDeployment failed (mentioned above).  Every time, you need to
see the name of the SoftwareDeployment which failed, figure out if it
failed on one or all of the servers, then look at the stderr for clues.

2. A server failed to build (OS::Nova::Server resource is FAILED), here we
need to check both nova and ironic, looking first to see if ironic has the
node(s) in the wrong state for scheduling (e.g nova gave us a no valid
host error), and then if they are OK in ironic, do nova show on the failed
host to see the reason nova gives us for it failing to go ACTIVE.

3. A stack timeout happened.  IIRC when this happens, we currently fail
with an obscure keystone related backtrace due to the token expiring.  We
should instead catch this error and show the heat stack status_reason,
which should say clearly the stack timed out.

If we could just make these three cases really clear and easy to debug, I
think things would be much better (IME the above are a high proportion of
all failures), but I'm sure folks can come up with other ideas to add to
the list.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Jeremy Stanley
On 2015-11-30 20:49:46 +1100 (+1100), Joshua Hesketh wrote:
> Freenode are currently experiencing a severe DDoS attack that are
> having an effect on our bots. As such the meetbot, irc logging and
> gerrit watcher are interminably available.
> 
> We expect the bots to resume their normal function once Freenode
> has recovered. For now, meetings may have to be postponed or
> minuted by hand.

I'm not sure if the attacks on Freenode have subsided yet, but I was
finally able to get the "openstack" meetbot reconnected around 15:45
UTC and it's seemed stable for half hour since. That's not to say I
have much faith at the moment it'll stick around, but it's at least
an improvement.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] notification subteam meeting

2015-11-30 Thread Balázs Gibizer
Hi, 

The next meeting of the nova notification subteam will happen 2015-12-01 
Tuesday 20:00 UTC [1] on #openstack-meeting-alt on freenode 

Agenda:
- Status of the outstanding specs and code reviews
- Mid-cycle
- AOB

See you there.

Cheers,
Gibi

 [1] https://www.timeanddate.com/worldclock/fixedtime.html?iso=20151201T20 
 [2] https://wiki.openstack.org/wiki/Meetings/NovaNotification


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Ironic] Do we need to have a mid-cycle?

2015-11-30 Thread Anita Kuno
On 11/23/2015 01:00 PM, Jim Rollenhagen wrote:
> On Mon, Nov 16, 2015 at 06:05:54AM -0800, Jim Rollenhagen wrote:
>>
>> Another idea I floated last week was to do a virtual midcycle of sorts.
>> Treat it like a normal midcycle in that everyone tells their management
>> "I'm out for 3-4 days for the midcycle", but they don't travel anywhere.
>> We come up with an agenda, see if there's any planning/syncing work to
>> do, or if it's all just hacking on code/reviews.
>>
>> Then we can set up some hangouts (or similar) to get people in the same
>> "room" working on things. Time zones will get weird, but we tend to
>> split into smaller groups at the midcycle anyway; this is just more
>> timezone-aligned. We can also find windows where time zones overlap when
>> we want to go across those boundaries. Disclaimer: people may need to
>> work some weird hours to do this well.
>>
>> I think this might get a little bit bumpy, but if it goes relatively
>> well we can try to improve on it for the future. Worst case, it's a
>> total failure and is roughly equivalent to the "no midcycle" option.
> 
> Nobody has objected, so we're going to roll with this. See y'all there. :)
> 
> // jim
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

Listing your virtual sprint on the virtual sprints wikipage is helpful
to those folks who might not work on ironic daily to consider helping:
https://wiki.openstack.org/wiki/VirtualSprints

Thanks,
Anita.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [release] using reno for libraries

2015-11-30 Thread Dmitry Tantsur

On 11/30/2015 05:14 PM, Doug Hellmann wrote:

Excerpts from Dmitry Tantsur's message of 2015-11-30 10:06:25 +0100:

On 11/28/2015 02:48 PM, Doug Hellmann wrote:

Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:

Liaisons,

We're making good progress on adding reno to service projects as
we head to the Mitaka-1 milestone. Thank you!

We also need to add reno to all of the other deliverables with
changes that might affect deployers. That means clients and other
libraries, SDKs, etc. with configuration options or where releases
can change deployment behavior in some way. Now that most teams
have been through this conversion once, it should be easy to replicate
for the other repositories in a similar way.

Libraries have 2 audiences for release notes: developers consuming
the library and deployers pushing out new versions of the libraries.
To separate the notes for the two audiences, and avoid doing manually
something that we have been doing automatically, we can use reno
just for deployer release notes (changes in support for options,
drivers, etc.). That means the library repositories that need reno
should have it configured just like for the service projects, with
the separate jobs and a publishing location different from their
existing developer documentation. The developer docs can continue
to include notes for the developer audience.


I've had a couple of questions about this split for release notes. The
intent is for developer-focused notes to continue to come from commit
messages and in-tree documentation, while using reno for new and
additional deployer-focused communication. Most commits to libraries
won't need reno release notes.


This looks like unnecessary overcomplication. Why not use such a
convenient tool for both kinds of release notes instead of having us
invent and maintain one more place to put release notes, now for


In the past we have had rudimentary release notes and changelogs
for developers to read based on the git commit messages. Since
deployers and developers care about different things, we don't want
to make either group sift through the notes meant for the other.
So, we publish notes in different ways.


Hmm, so maybe for small libraries with few changes it's still fine to 
publish them together, what do you think?




The thing that is new here is publishing release notes for changes
in libraries that deployers need to know about. While the Oslo code
was in the incubator, and being copied into applications, it was
possible to detect deployer-focused changes like new or deprecated
configuration options in the application and put the notes there.
Using shared libraries means those changes can happen without
application developers being aware of them, so the library maintainers
need to be publishing notes. Using reno for those notes is consistent
with the way they are handled in the applications, so we're extending
one tool to more repositories.


developers? It's already not so easy to explain reno to newcomers, this
idea makes it even harder...


Can you tell me more about the difficulty you've had? I would like to
improve the documentation for reno and for how we use it.


Usually people are stuck at the "how do I do this at all" stage :) we've 
even added it to the ironic developer FAQ. As to me, the official reno 
documentation is nice enough (but see below), maybe people are not aware 
of it.


Another "issue" (at least for our newcomers) with reno docs is that 
http://docs.openstack.org/developer/reno/usage.html#generating-a-report 
mentions the "reno report" command which is not something we all 
actually use, we use these "tox -ereleasenotes" command. What is worse, 
this command (I guess it's by design) does not catch release note files 
that are just created locally. It took me time to figure out that I have 
to commit release notes before "tox -ereleasenotes" would show them in 
the rendered HTML.


Finally, people are confused by how our release note jobs handle 
branches. E.g. ironic-inspector release notes [1] currently seem to show 
release notes from stable/liberty (judging by the version), so no 
current items [2] are shown.


[1] http://docs.openstack.org/releasenotes/ironic-inspector/unreleased.html
[2] for example 
http://docs-draft.openstack.org/18/250418/2/gate/gate-ironic-inspector-releasenotes/f0b9363//releasenotes/build/html/unreleased.html




Doug





Doug



After we start using reno for libraries, the release announcement
email tool will be updated to use those same notes to build the
message in addition to looking at the git change log. This will be
a big step toward unifying the release process for services and
libraries, and will allow us to make progress on completing the
automation work we have planned for this cycle.

It's not necessary to add reno to the liberty branch for library
projects, since we tend to backport far fewer changes to libraries.
If you maintain a library that does see a lot of backports, by all
means 

Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Dmitry Tantsur

On 11/30/2015 04:19 PM, Derek Higgins wrote:

Hi All,

 A few months tripleo switch from its devtest based CI to one that
was based on instack. Before doing this we anticipated disruption in the
ci jobs and removed them from non tripleo projects.

 We'd like to investigate adding it back to heat and ironic as these
are the two projects where we find our ci provides the most value. But
we can only do this if the results from the job are treated as voting.

 In the past most of the non tripleo projects tended to ignore the
results from the tripleo job as it wasn't unusual for the job to broken
for days at a time. The thing is, ignoring the results of the job is the
reason (the majority of the time) it was broken in the first place.
 To decrease the number of breakages we are now no longer running
master code for everything (for the non tripleo projects we bump the
versions we use periodically if they are working). I believe with this
model the CI jobs we run have become a lot more reliable, there are
still breakages but far less frequently.

What I proposing is we add at least one of our tripleo jobs back to both
heat and ironic (and other projects associated with them e.g. clients,
ironicinspector etc..), tripleo will switch to running latest master of
those repositories and the cores approving on those projects should wait
for a passing CI jobs before hitting approve. So how do people feel
about doing this? can we give it a go? A couple of people have already
expressed an interest in doing this but I'd like to make sure were all
in agreement before switching it on.


I'm one of these "people", so definitely +1 here.

By the way, is it possible to NOT run tripleo-ci on changes touching 
only tests and docs? We do the same for our devstack jobs, it saves some 
infra resources.




thanks,
Derek.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [mistral] Team meeting minutes - 11/30/2015

2015-11-30 Thread Anastasia Kuznetsova
Thanks everyone for joining us!

Meeting minutes:
http://eavesdrop.openstack.org/meetings/mistral/2015/mistral.2015-11-30-16.01.html
Meeting log:
http://eavesdrop.openstack.org/meetings/mistral/2015/mistral.2015-11-30-16.01.log.html

-- 
Best regards,
Anastasia Kuznetsova
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [keystone] Removing functionality that was deprecated in Kilo and upcoming deprecated functionality in Mitaka

2015-11-30 Thread Steve Martinelli

In the Mitaka release, the keystone team will be removing functionality
that was marked for deprecation in Kilo, and marking certain functions as
deprecated in Mitaka (that may be removed in at least 2 cycles).

removing deprecated functionality
=

This is not a full list, but these are by and large the most contentious
topics.

* Eventlet support: This was marked as deprecated back in Kilo and is
currently scheduled to be removed in Mitaka in favor of running keystone in
a WSGI server. This is currently how we test keystone in the gate, and
based on the feedback we received at the summit, a lot of folks have moved
to running keystone under Apache since we’ve announced this change.
OpenStack's CI is configured to mainly test using this deployment model.
See [0] for when we started to issue warnings.

* Using LDAP to store assignment data: Like eventlet support, this feature
was also deprecated in Kilo and scheduled to be removed in Mitaka. To store
assignment data (role assignments) we suggest using an SQL based backend
rather than LDAP. See [1] for when we started to issue warnings.

* Using LDAP to store project and domain data: The same as above, see [2]
for when we started to issue warnings.

* for a complete list:
https://blueprints.launchpad.net/keystone/+spec/removed-as-of-mitaka

functions deprecated as of mitaka
=

The following will adhere to the TC’s new standard on deprecating
functionality [3].

* LDAP write support for identity: We suggest simply not writing to LDAP
for users and groups, this effectively makes create, delete and update of
LDAP users and groups a no-op. It will be removed in the O release.

* PKI tokens: We suggest using UUID or fernet tokens instead. The PKI token
format has had issues with security and causes problems with both horizon
and swift when the token contains an excessively large service catalog. It
will be removed in the O release.

* v2.0 of our API: Lastly, the keystone team recommends using v3 of our
Identity API. We have had the intention of deprecating v2.0 for a while
(since Juno actually), and have finally decided to formally deprecate v2.0.
OpenStack’s CI runs successful v3 only jobs, there is complete feature
parity with v2.0, and we feel the CLI exposed via openstackclient is mature
enough to say with certainty that we can deprecate v2.0. It will be around
for at least FOUR releases, with the authentication routes
(POST /auth/tokens) potentially sticking around for longer.

* for a complete list:
https://blueprints.launchpad.net/keystone/+spec/deprecated-as-of-mitaka


If you have ANY concern about the following, please speak up now and let us
know!


Thanks!

Steve Martinelli
OpenStack Keystone Project Team Lead


[0]
https://github.com/openstack/keystone/blob/b475040636ccc954949e6372a60dd86845644611/keystone/server/eventlet.py#L77-L80
[1]
https://github.com/openstack/keystone/blob/28a30f53a6c0d4e84d60795e08f137e8194abbe9/keystone/assignment/backends/ldap.py#L34
[2]
https://github.com/openstack/keystone/blob/28a30f53a6c0d4e84d60795e08f137e8194abbe9/keystone/resource/backends/ldap.py#L36-L39

[3]
http://governance.openstack.org/reference/tags/assert_follows-standard-deprecation.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

2015-11-30 Thread Anita Kuno
On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:
> Hi,
> 
> 
> 
>  This is to announce that  we have  setup  a  Third Party CI environment
> for Proliant iLO Drivers. The results will be posted  under "HP Proliant CI
> check" section in Non-voting mode.   We will be  running the basic deploy
> tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We will
> first  pursue to make the results consistent and over a period of time we
> will try to promote it to voting mode.
> 
> 
> 
>For more information check the Wiki:
> https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci ,
> for any issues please contact ilo_driv...@groups.ext.hpe.com
> 
> 
> 
> 
> 
> Thanks & Regards,
> 
> Gururaja Grandhi
> 
> R Project Manager
> 
> HPE Proliant  Ironic  Project
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

Please do not post announcements to the mailing list about the existence
of your third party ci system.

Ensure your third party ci system is listed here:
https://wiki.openstack.org/wiki/ThirdPartySystems (there are
instructions on the bottom of the page) as well as fill out a template
on your system so that folks can find your third party ci system the
same as all other third party ci systems.

Ensure you are familiar with the requirements for third party systems
listed here:
http://docs.openstack.org/infra/system-config/third_party.html#requirements

Thank you,
Anita.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Lucas Alvares Gomes
I'm +1 for re-enabling it in Ironic. I think ironic plays a big role
in the tripleo puzzle so we should be aware of breakages in the
tripleo project.

On Mon, Nov 30, 2015 at 3:19 PM, Derek Higgins  wrote:
> Hi All,
>
> A few months tripleo switch from its devtest based CI to one that was
> based on instack. Before doing this we anticipated disruption in the ci jobs
> and removed them from non tripleo projects.
>
> We'd like to investigate adding it back to heat and ironic as these are
> the two projects where we find our ci provides the most value. But we can
> only do this if the results from the job are treated as voting.
>
> In the past most of the non tripleo projects tended to ignore the
> results from the tripleo job as it wasn't unusual for the job to broken for
> days at a time. The thing is, ignoring the results of the job is the reason
> (the majority of the time) it was broken in the first place.
> To decrease the number of breakages we are now no longer running master
> code for everything (for the non tripleo projects we bump the versions we
> use periodically if they are working). I believe with this model the CI jobs
> we run have become a lot more reliable, there are still breakages but far
> less frequently.
>
> What I proposing is we add at least one of our tripleo jobs back to both
> heat and ironic (and other projects associated with them e.g. clients,
> ironicinspector etc..), tripleo will switch to running latest master of
> those repositories and the cores approving on those projects should wait for
> a passing CI jobs before hitting approve. So how do people feel about doing
> this? can we give it a go? A couple of people have already expressed an
> interest in doing this but I'd like to make sure were all in agreement
> before switching it on.
>
> thanks,
> Derek.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

2015-11-30 Thread Dmitry Tantsur

On 11/30/2015 05:34 PM, Anita Kuno wrote:

On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:

Hi,



  This is to announce that  we have  setup  a  Third Party CI environment
for Proliant iLO Drivers. The results will be posted  under "HP Proliant CI
check" section in Non-voting mode.   We will be  running the basic deploy
tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We will
first  pursue to make the results consistent and over a period of time we
will try to promote it to voting mode.



For more information check the Wiki:
https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci ,
for any issues please contact ilo_driv...@groups.ext.hpe.com





Thanks & Regards,

Gururaja Grandhi

R Project Manager

HPE Proliant  Ironic  Project



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Please do not post announcements to the mailing list about the existence
of your third party ci system.


Could you please explain why? As a developer I appreciated this post.



Ensure your third party ci system is listed here:
https://wiki.openstack.org/wiki/ThirdPartySystems (there are
instructions on the bottom of the page) as well as fill out a template
on your system so that folks can find your third party ci system the
same as all other third party ci systems.


Wiki is not an announcement FWIW.



Ensure you are familiar with the requirements for third party systems
listed here:
http://docs.openstack.org/infra/system-config/third_party.html#requirements

Thank you,
Anita.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Encouraging first-time contributors through bug tags/reviews

2015-11-30 Thread Doug Hellmann
Excerpts from sean roberts's message of 2015-11-30 07:57:54 -0800:
> How about:
> First timers assign a bug to a mentor and the mentor takes responsibility
> for the first timer learning from the bug to completion.

That would mean the learning process is different from what we want the
regular process to be.

If the problem is identifying "In Progress" bugs that are actually not
being worked on, then let's figure out a way to make that easier.
sdague's point about the auto-abandon process may help. We could query
gerrit for "stale" reviews that would have met the old abandon
requirements and that refer to bugs, for example. Using that
information, someone could follow-up with the patch owner to see if it
is actually abandoned, before changing the bug status or encouraging the
owner to abandon the patch.

> 
> Per project, a few people volunteer themselves as mentors. As easy as
> responding to [project][mentor] emails.
> 
> On Monday, November 30, 2015, Sean Dague  wrote:
> 
> > On 11/25/2015 03:22 PM, Shamail wrote:
> > > Hi,
> > >
> > >> On Nov 25, 2015, at 11:05 PM, Doug Hellmann  > > wrote:
> > >>
> > >> Excerpts from Shamail Tahir's message of 2015-11-25 09:15:54 -0500:
> > >>> Hi everyone,
> > >>>
> > >>> Andrew Mitry recently shared a medium post[1] by Kent C. Dobbs which
> > >>> discusses how one open-source project is encouraging contributions by
> > new
> > >>> open-source contributors through a combination of a special tag (which
> > is
> > >>> associated with work that is needed but can only be completed by
> > someone
> > >>> who is a first-time contributor) and helpful comments in the review
> > phase
> > >>> to ensure the contribution(s) eventually get merged.
> > >>>
> > >>> While reading the article, I immediately thought about our
> > >>> low-hanging-fruit bug tag which is used for a very similar purpose in
> > "bug
> > >>> fixing" section of  the "how to contribute" page[2].  The
> > low-hanging-fruit
> > >>> tag is used to identify items that are generally suitable for
> > first-time or
> > >>> beginner contributors but, in reality, anyone can pick them up.
> > >>>
> > >>> I wanted to propose a new tag (or even changing the, existing,
> > low-hanging
> > >>> fruit tag) that would identify items that we are reserving for
> > first-time
> > >>> OpenStack contributors (e.g. a patch-set for the item submitted by
> > someone
> > >>> who is not a first time contributor would be rejected)... The same
> > article
> > >>> that Andrew shared mentions using an "up-for-grabs" tag which also
> > >>> populates the items at up-for-grabs[3] (a site where people looking to
> > >>> start working on open-source projects see entry-level items from
> > multiple
> > >>> projects).  If we move forward with an exclusive tag for first-timers
> > then
> > >>> it would be nice if we could use the up-for-grabs tag so that OpenStack
> > >>> also shows up on the list too.  Please let me know if this change
> > should be
> > >>> proposed elsewhere, the tags are maintained in launchpad and the wiki I
> > >>> found related to bug tags[4] didn't indicate a procedure for
> > submitting a
> > >>> change proposal.
> > >>
> > >> I like the idea of making bugs suitable for first-timers more
> > >> discoverable. I'm not sure we need to *reserve* any bugs for any class
> > >> of contributor. What benefit do you think that provides?
> > > I would have to defer to additional feedback here...
> > >
> > > My own perspective from when I was doing my first contribution is that
> > it was hard to find active "low-hanging-fruit" items.  Most were already
> > work-in-progress or assigned.
> >
> > This was a direct consequence of us dropping the auto-abandoning of old
> > code reviews in gerrit. When a review is abandoned the bug is flipped
> > back to New instead of In Progress.
> >
> > I found quite often people go and gobble up bugs assigning them to
> > themselves, but don't make real progress on them. Then new contributors
> > show up, and don't work on any of those issues because our tools say
> > someone is already on top of it.
> >
> > -Sean
> >
> > --
> > Sean Dague
> > http://dague.net
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

2015-11-30 Thread Gururaj Grandhi
Hi,



 This is to announce that  we have  setup  a  Third Party CI environment
for Proliant iLO Drivers. The results will be posted  under "HP Proliant CI
check" section in Non-voting mode.   We will be  running the basic deploy
tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We will
first  pursue to make the results consistent and over a period of time we
will try to promote it to voting mode.



   For more information check the Wiki:
https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci ,
for any issues please contact ilo_driv...@groups.ext.hpe.com





Thanks & Regards,

Gururaja Grandhi

R Project Manager

HPE Proliant  Ironic  Project
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread Walter A. Boring IV
As a side note to the DR discussion here, there was a session in Tokyo 
that talked about a new

DR project called Smaug.   You can see their mission statement here:
https://launchpad.net/smaug

https://github.com/openstack/smaug

There is another service in the making called DRagon:
https://www.youtube.com/watch?v=upCzuFnswtw
http://www.slideshare.net/AlonMarx/dragon-and-cinder-v-brownbag-54639869

Yes that's 2 DR like service starting in OpenStack that are related to 
dragons.


Walt



Sean and Michal,

In fact, there is a reason that I ask this question. Recently I have a
confusion about if cinder should provide the ability of Disaster
Recovery to storage resources, like volume. I mean we have volume
replication v1, but for DR, specially DR between two independent
OpenStack sites(production and DR site), I feel we still need more
features to support it, for example consistency group for replication,
etc. I'm not sure if those features belong in Cinder or some new
project for DR.

BR
WangHao

2015-11-30 3:02 GMT+08:00 Sean McGinnis :

On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:

On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:

Hi guys,

I notice nova have a clarification of project scope:
http://docs.openstack.org/developer/nova/project_scope.html

I want to find cinder's, but failed,  do you know where to find it?

It's important to let developers know what feature should be
introduced into cinder and what shouldn't.

BR
Wang Hao

I believe Nova team needed to formalize the scop to have an explanation
for all the "this doesn't belong in Nova" comments on feature requests.
Does Cinder suffer from similar problems? From my perspective it's not
critically needed.

I agree. I haven't seen a need for something like that with Cinder. Wang
Hao, is there a reason you feel you need that?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Derek Higgins



On 30/11/15 17:03, Dmitry Tantsur wrote:

On 11/30/2015 04:19 PM, Derek Higgins wrote:

Hi All,

 A few months tripleo switch from its devtest based CI to one that
was based on instack. Before doing this we anticipated disruption in the
ci jobs and removed them from non tripleo projects.

 We'd like to investigate adding it back to heat and ironic as these
are the two projects where we find our ci provides the most value. But
we can only do this if the results from the job are treated as voting.

 In the past most of the non tripleo projects tended to ignore the
results from the tripleo job as it wasn't unusual for the job to broken
for days at a time. The thing is, ignoring the results of the job is the
reason (the majority of the time) it was broken in the first place.
 To decrease the number of breakages we are now no longer running
master code for everything (for the non tripleo projects we bump the
versions we use periodically if they are working). I believe with this
model the CI jobs we run have become a lot more reliable, there are
still breakages but far less frequently.

What I proposing is we add at least one of our tripleo jobs back to both
heat and ironic (and other projects associated with them e.g. clients,
ironicinspector etc..), tripleo will switch to running latest master of
those repositories and the cores approving on those projects should wait
for a passing CI jobs before hitting approve. So how do people feel
about doing this? can we give it a go? A couple of people have already
expressed an interest in doing this but I'd like to make sure were all
in agreement before switching it on.


I'm one of these "people", so definitely +1 here.

By the way, is it possible to NOT run tripleo-ci on changes touching
only tests and docs? We do the same for our devstack jobs, it saves some
infra resources.
We don't do it currently, but I'm sure we could and it sounds like a 
good idea to me.






thanks,
Derek.

__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

2015-11-30 Thread Anita Kuno
On 11/30/2015 12:17 PM, Dmitry Tantsur wrote:
> On 11/30/2015 05:34 PM, Anita Kuno wrote:
>> On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:
>>> Hi,
>>>
>>>
>>>
>>>   This is to announce that  we have  setup  a  Third Party CI
>>> environment
>>> for Proliant iLO Drivers. The results will be posted  under "HP
>>> Proliant CI
>>> check" section in Non-voting mode.   We will be  running the basic
>>> deploy
>>> tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We
>>> will
>>> first  pursue to make the results consistent and over a period of
>>> time we
>>> will try to promote it to voting mode.
>>>
>>>
>>>
>>> For more information check the Wiki:
>>> https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci
>>> ,
>>> for any issues please contact ilo_driv...@groups.ext.hpe.com
>>>
>>>
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> Gururaja Grandhi
>>>
>>> R Project Manager
>>>
>>> HPE Proliant  Ironic  Project
>>>
>>>
>>>
>>> __
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> Please do not post announcements to the mailing list about the existence
>> of your third party ci system.
> 
> Could you please explain why? As a developer I appreciated this post.
> 
>>
>> Ensure your third party ci system is listed here:
>> https://wiki.openstack.org/wiki/ThirdPartySystems (there are
>> instructions on the bottom of the page) as well as fill out a template
>> on your system so that folks can find your third party ci system the
>> same as all other third party ci systems.
> 
> Wiki is not an announcement FWIW.

If Ironic wants to hear about announced drivers they have agreed to do
so as part of their weekly irc meeting:
2015-11-30T17:19:55   I think it is reasonable for each
driver team, if they want to announce it in the meeting, to do so on the
whiteboard section for their driver. we'll all see that in the weekly
meeting
2015-11-30T17:20:08   but it will avoid spamming the whole
openstack list

http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2015-11-30.log

Thank you,
Anita.

> 
>>
>> Ensure you are familiar with the requirements for third party systems
>> listed here:
>> http://docs.openstack.org/infra/system-config/third_party.html#requirements
>>
>>
>> Thank you,
>> Anita.
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Derek Higgins

Hi All,

A few months tripleo switch from its devtest based CI to one that 
was based on instack. Before doing this we anticipated disruption in the 
ci jobs and removed them from non tripleo projects.


We'd like to investigate adding it back to heat and ironic as these 
are the two projects where we find our ci provides the most value. But 
we can only do this if the results from the job are treated as voting.


In the past most of the non tripleo projects tended to ignore the 
results from the tripleo job as it wasn't unusual for the job to broken 
for days at a time. The thing is, ignoring the results of the job is the 
reason (the majority of the time) it was broken in the first place.
To decrease the number of breakages we are now no longer running 
master code for everything (for the non tripleo projects we bump the 
versions we use periodically if they are working). I believe with this 
model the CI jobs we run have become a lot more reliable, there are 
still breakages but far less frequently.


What I proposing is we add at least one of our tripleo jobs back to both 
heat and ironic (and other projects associated with them e.g. clients, 
ironicinspector etc..), tripleo will switch to running latest master of 
those repositories and the cores approving on those projects should wait 
for a passing CI jobs before hitting approve. So how do people feel 
about doing this? can we give it a go? A couple of people have already 
expressed an interest in doing this but I'd like to make sure were all 
in agreement before switching it on.


thanks,
Derek.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Encouraging first-time contributors through bug tags/reviews

2015-11-30 Thread sean roberts
How about:
First timers assign a bug to a mentor and the mentor takes responsibility
for the first timer learning from the bug to completion.

Per project, a few people volunteer themselves as mentors. As easy as
responding to [project][mentor] emails.

On Monday, November 30, 2015, Sean Dague  wrote:

> On 11/25/2015 03:22 PM, Shamail wrote:
> > Hi,
> >
> >> On Nov 25, 2015, at 11:05 PM, Doug Hellmann  > wrote:
> >>
> >> Excerpts from Shamail Tahir's message of 2015-11-25 09:15:54 -0500:
> >>> Hi everyone,
> >>>
> >>> Andrew Mitry recently shared a medium post[1] by Kent C. Dobbs which
> >>> discusses how one open-source project is encouraging contributions by
> new
> >>> open-source contributors through a combination of a special tag (which
> is
> >>> associated with work that is needed but can only be completed by
> someone
> >>> who is a first-time contributor) and helpful comments in the review
> phase
> >>> to ensure the contribution(s) eventually get merged.
> >>>
> >>> While reading the article, I immediately thought about our
> >>> low-hanging-fruit bug tag which is used for a very similar purpose in
> "bug
> >>> fixing" section of  the "how to contribute" page[2].  The
> low-hanging-fruit
> >>> tag is used to identify items that are generally suitable for
> first-time or
> >>> beginner contributors but, in reality, anyone can pick them up.
> >>>
> >>> I wanted to propose a new tag (or even changing the, existing,
> low-hanging
> >>> fruit tag) that would identify items that we are reserving for
> first-time
> >>> OpenStack contributors (e.g. a patch-set for the item submitted by
> someone
> >>> who is not a first time contributor would be rejected)... The same
> article
> >>> that Andrew shared mentions using an "up-for-grabs" tag which also
> >>> populates the items at up-for-grabs[3] (a site where people looking to
> >>> start working on open-source projects see entry-level items from
> multiple
> >>> projects).  If we move forward with an exclusive tag for first-timers
> then
> >>> it would be nice if we could use the up-for-grabs tag so that OpenStack
> >>> also shows up on the list too.  Please let me know if this change
> should be
> >>> proposed elsewhere, the tags are maintained in launchpad and the wiki I
> >>> found related to bug tags[4] didn't indicate a procedure for
> submitting a
> >>> change proposal.
> >>
> >> I like the idea of making bugs suitable for first-timers more
> >> discoverable. I'm not sure we need to *reserve* any bugs for any class
> >> of contributor. What benefit do you think that provides?
> > I would have to defer to additional feedback here...
> >
> > My own perspective from when I was doing my first contribution is that
> it was hard to find active "low-hanging-fruit" items.  Most were already
> work-in-progress or assigned.
>
> This was a direct consequence of us dropping the auto-abandoning of old
> code reviews in gerrit. When a review is abandoned the bug is flipped
> back to New instead of In Progress.
>
> I found quite often people go and gobble up bugs assigning them to
> themselves, but don't make real progress on them. Then new contributors
> show up, and don't work on any of those issues because our tools say
> someone is already on top of it.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


-- 
~sean
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [release] using reno for libraries

2015-11-30 Thread Doug Hellmann
Excerpts from Dmitry Tantsur's message of 2015-11-30 10:06:25 +0100:
> On 11/28/2015 02:48 PM, Doug Hellmann wrote:
> > Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:
> >> Liaisons,
> >>
> >> We're making good progress on adding reno to service projects as
> >> we head to the Mitaka-1 milestone. Thank you!
> >>
> >> We also need to add reno to all of the other deliverables with
> >> changes that might affect deployers. That means clients and other
> >> libraries, SDKs, etc. with configuration options or where releases
> >> can change deployment behavior in some way. Now that most teams
> >> have been through this conversion once, it should be easy to replicate
> >> for the other repositories in a similar way.
> >>
> >> Libraries have 2 audiences for release notes: developers consuming
> >> the library and deployers pushing out new versions of the libraries.
> >> To separate the notes for the two audiences, and avoid doing manually
> >> something that we have been doing automatically, we can use reno
> >> just for deployer release notes (changes in support for options,
> >> drivers, etc.). That means the library repositories that need reno
> >> should have it configured just like for the service projects, with
> >> the separate jobs and a publishing location different from their
> >> existing developer documentation. The developer docs can continue
> >> to include notes for the developer audience.
> >
> > I've had a couple of questions about this split for release notes. The
> > intent is for developer-focused notes to continue to come from commit
> > messages and in-tree documentation, while using reno for new and
> > additional deployer-focused communication. Most commits to libraries
> > won't need reno release notes.
> 
> This looks like unnecessary overcomplication. Why not use such a 
> convenient tool for both kinds of release notes instead of having us 
> invent and maintain one more place to put release notes, now for 

In the past we have had rudimentary release notes and changelogs
for developers to read based on the git commit messages. Since
deployers and developers care about different things, we don't want
to make either group sift through the notes meant for the other.
So, we publish notes in different ways.

The thing that is new here is publishing release notes for changes
in libraries that deployers need to know about. While the Oslo code
was in the incubator, and being copied into applications, it was
possible to detect deployer-focused changes like new or deprecated
configuration options in the application and put the notes there.
Using shared libraries means those changes can happen without
application developers being aware of them, so the library maintainers
need to be publishing notes. Using reno for those notes is consistent
with the way they are handled in the applications, so we're extending
one tool to more repositories.

> developers? It's already not so easy to explain reno to newcomers, this 
> idea makes it even harder...

Can you tell me more about the difficulty you've had? I would like to
improve the documentation for reno and for how we use it.

Doug

> 
> >
> > Doug
> >
> >>
> >> After we start using reno for libraries, the release announcement
> >> email tool will be updated to use those same notes to build the
> >> message in addition to looking at the git change log. This will be
> >> a big step toward unifying the release process for services and
> >> libraries, and will allow us to make progress on completing the
> >> automation work we have planned for this cycle.
> >>
> >> It's not necessary to add reno to the liberty branch for library
> >> projects, since we tend to backport far fewer changes to libraries.
> >> If you maintain a library that does see a lot of backports, by all
> >> means go ahead and add reno, but it's not a requirement. If you do
> >> set up multiple branches, make sure you have one page that uses the
> >> release-notes directive without specifing a branch, as in the
> >> oslo.config example, to build notes for the "current" branch to get
> >> releases from master and to serve as a test for rendering notes
> >> added to stable branches.
> >>
> >> Thanks,
> >> Doug
> >>
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Glance] PSA: Today's Artifacts meeting canceled.

2015-11-30 Thread Nikhil Komawar
Hi all,

There was some bot issue this morning and all the meeting commands
weren't working hence.

We canceled the Artifacts meeting after ~20 mins however, there was a
lot of informal chatting on the #openstack-meeting-4 channel during the
wait (no significant decisions taken, and there are plans to further
improve the meeting and other collaborations).

-- 

Thanks,
Nikhil


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Performance] Filling performance team working items list and taking part in their resolving

2015-11-30 Thread Matt Riedemann



On 11/27/2015 3:54 AM, Dina Belova wrote:

Hey OpenStack devs and operators!

Folks, I would like to share list of working items Performance Team is
currently having in the backlog -
https://etherpad.openstack.org/p/perf-zoom-zoom [Work Items to grab]
section. I'm really encouraging you to fill it with concrete pieces of
work you think will be useful and take part in the
development/investigation by assigning some of them to yourself and
working on them :)

Cheers,
Dina


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



One of the work items was to enable the nova metadata service in a large 
ops job. There are two voting large ops jobs in nova, one runs with 
nova-network and the other runs with neutron. Besides those differences, 
I'm not sure if there is any other difference in the jobs. So I guess 
we'd just need to pick which one runs the nova-api-metadata service 
rather than using config drive. The only other job I know of that runs 
that is the postgres job and that runs nova-network, so I'd say we turn 
on n-api-meta in the neutron large ops job.


Are there any issues with that?

--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [horizon] Supporting Django 1.9

2015-11-30 Thread Thomas Goirand
On 11/29/2015 08:59 PM, Rob Cresswell (rcresswe) wrote:
> https://blueprints.launchpad.net/horizon/+spec/drop-dj17
> 
> 
> This is where changes are currently being tracked. I don’t quite
> understand why these would be back ported; they would break Liberty with
> 1.7. Perhaps we can discuss on IRC tomorrow?
> 
> Rob

Before Mitaka is out, Sid will hold Horizon from Liberty. I don't care
having breakage for Django 1.7 (Sid has 1.8 already), but what I need is
having support for Django 1.9. That's what I want to backport.

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Neutron] Automatically generate sample configuration files (oslo config generator)

2015-11-30 Thread Martin Hickey

Hi,

The generation of Neutron configuration files using the oslo config
generator is being introduced in Mitaka as part of blueprint [1] and bug
[2]. This is broken down into two phases: Neutron core and Neutron *aaS
projects.

The first part of the first phase is the generation of sample core
configuration files in patch [3]. The next part is the removal of the
sample configuration files from the core repository in patch [4]. Patch [3]
and [5] make the static sample configuration files in the Neutron
repository redundant.  Developers should no longer update these
configuration files as  they will be removed in patch[4]. Refer to the
Neutron contribution guide for more details.

The final phase of the blueprint is the update of the Neutron *aaS projects
for configuration generation. These commits will be associated to bug [2].

It is advised that Neutron sub-projects do not keep their configuration
files in their respective trees and instead generate them using a similar
approach as Neutron does. Operators should refer to the release notes for
more details.

[1]
https://blueprints.launchpad.net/neutron/+spec/autogen-neutron-conf-file
[2] https://bugs.launchpad.net/neutron/+bug/1199963
[3] https://review.openstack.org/#/c/204206/
[4] https://review.openstack.org/#/c/251348/
[5] https://review.openstack.org/#/c/204722/

Regards,
Martin


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Clark, Jay
Can't connect either. Dead in the water

Regards,
Jay Clark
Sr. OpenStack Deployment Engineer
E: jason.t.cl...@hpe.com
H: 919.341.4670
M: 919.345.1127
IRC (freenode): jasondotstar


From: lichen.hangzhou [lichen.hangz...@gmail.com]
Sent: Monday, November 30, 2015 9:17 AM
To: OpenStack Development Mailing List (not for usage questions)
Cc: openstack-in...@lists.openstack.org
Subject: Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

Can't connect +1 and web client do not work  :(

-chen

At 2015-11-30 22:08:12, "Hinds, Luke (Nokia - GB/Bristol)" 
 wrote:
Me too.  It is possible to get on using the web client though;  
https://webchat.freenode.net/ .

On Mon, 2015-11-30 at 14:00 +, EXT Dugger, Donald D wrote:
I can’t even connect to the IRC server at all, can others get it?

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786

From: Joshua Hesketh 
[mailto:joshua.hesk...@gmail.com]
Sent: Monday, November 30, 2015 2:50 AM
To: OpenStack Development Mailing List (not for usage questions) 
>; 
openstack-infra 
>
Subject: [openstack-dev] IRC Bot issues

Hi all,
Freenode are currently experiencing a severe DDoS attack that are having an 
effect on our bots. As such the meetbot, irc logging and gerrit watcher are 
interminably available.

We expect the bots to resume their normal function once Freenode has recovered. 
For now, meetings may have to be postponed or minuted by hand.
Cheers,
Josh

___
OpenStack-Infra mailing list
openstack-in...@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread yang, xing
Hi Wang Hao,

Here¹s a Cinder spec in review on replicating a group of volumes:

https://review.openstack.org/#/c/229722/

It is a not an easy problem to solve.  Not that we should rush on this
problem, but we should start thinking about how to solve this as some
backends can only replicate a CG or a pool of volumes.

Thanks,
Xing



On 11/30/15, 4:51 AM, "hao wang"  wrote:

>Hi, Duncan
>
>2015-11-30 15:54 GMT+08:00 Duncan Thomas :
>> Hi WangHao
>>
>> This was quite thoroughly discussed during the early discussions on
>> replication. The general statement was 'not yet'. Getting any kind of
>> workable replication API has proven to be very, very difficult to get
>>right
>> - we won't know for another full cycle whether we've actually gotten it
>> somewhere near right, as operators start to deploy it. Piling more
>>feature
>> in the replication API before a) it has been used by operators and b)
>> storage vendors have implemented what we already has would IMO be a
>>mistake.
>
>I agree with you, in my mind, using replication what we have is first
>thing we should done,
>improve it much better is second thing, and then we will add another
>new features
>one by one stably.
>
>> None of this means that more DR interfaces don't belong in cinder, just
>>that
>> getting them right, getting them universal and getting them useful is
>>quite
>> a hard problem, and not one we should be in a rush to solve.
>>Particularly as
>> DR and replication is still a niche area of cinder, and we still have
>>major
>> issues in our basic functionality.
>
>Yes, this will convince me about DR in Cinder, very clearly, thanks.
>>
>> On 30 November 2015 at 03:45, hao wang  wrote:
>>>
>>> Sean and Michal,
>>>
>>> In fact, there is a reason that I ask this question. Recently I have a
>>> confusion about if cinder should provide the ability of Disaster
>>> Recovery to storage resources, like volume. I mean we have volume
>>> replication v1, but for DR, specially DR between two independent
>>> OpenStack sites(production and DR site), I feel we still need more
>>> features to support it, for example consistency group for replication,
>>> etc. I'm not sure if those features belong in Cinder or some new
>>> project for DR.
>>>
>>> BR
>>> WangHao
>>>
>>> 2015-11-30 3:02 GMT+08:00 Sean McGinnis :
>>> > On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
>>> >> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
>>> >> > Hi guys,
>>> >> >
>>> >> > I notice nova have a clarification of project scope:
>>> >> > http://docs.openstack.org/developer/nova/project_scope.html
>>> >> >
>>> >> > I want to find cinder's, but failed,  do you know where to find
>>>it?
>>> >> >
>>> >> > It's important to let developers know what feature should be
>>> >> > introduced into cinder and what shouldn't.
>>> >> >
>>> >> > BR
>>> >> > Wang Hao
>>> >>
>>> >> I believe Nova team needed to formalize the scop to have an
>>>explanation
>>> >> for all the "this doesn't belong in Nova" comments on feature
>>>requests.
>>> >> Does Cinder suffer from similar problems? From my perspective it's
>>>not
>>> >> critically needed.
>>> >
>>> > I agree. I haven't seen a need for something like that with Cinder.
>>>Wang
>>> > Hao, is there a reason you feel you need that?
>>> >
>>> >
>>> >
>>> > 
>>>
>>>__
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe:
>>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> 
>>>
>>>__
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> --
>> --
>> Duncan Thomas
>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread Anita Kuno
On 11/29/2015 02:02 PM, Sean McGinnis wrote:
> On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
>> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
>>> Hi guys,
>>>
>>> I notice nova have a clarification of project scope:
>>> http://docs.openstack.org/developer/nova/project_scope.html
>>>
>>> I want to find cinder's, but failed,  do you know where to find it?
>>>
>>> It's important to let developers know what feature should be
>>> introduced into cinder and what shouldn't.
>>>
>>> BR
>>> Wang Hao
>>
>> I believe Nova team needed to formalize the scop to have an explanation
>> for all the "this doesn't belong in Nova" comments on feature requests.
>> Does Cinder suffer from similar problems? From my perspective it's not
>> critically needed.
> 
> I agree. I haven't seen a need for something like that with Cinder. Wang
> Hao, is there a reason you feel you need that?
> 

For reference here is the Cinder mission statement:
http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml#n273

All projects listed in the governance repository reference/projects.yaml
have a mission statement, I do encourage folks thinking about starting a
project to look at the mission statements here first as there may
already be an effort ongoing with which you can align your work.

Thanks Wang Hao,
Anita.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova]New Quota Subteam on Nova

2015-11-30 Thread melanie witt
On Nov 26, 2015, at 9:36, John Garbutt  wrote:

> A suggestion in the past, that I like, is creating a nova functional
> test that stress tests the quota code.
> 
> Hopefully that will be able to help reproduce the error.
> That should help prove if any proposed fix actually works.

+1, I think it's wise to get some data on the current state of quotas before 
choosing a redesign. IIRC, Joe Gordon described a test scenario he used to use 
to reproduce quota bugs locally, in one of the launchpad bugs. If we could 
automate something like that, we could use it to demonstrate how quotas 
currently behave during parallel requests and try things like disabling 
reservations. I also like the idea of being able to verify the effects of 
proposed fixes.

-melanie (irc: melwitt)







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [release] using reno for libraries

2015-11-30 Thread Doug Hellmann
Excerpts from Dmitry Tantsur's message of 2015-11-30 17:58:45 +0100:
> On 11/30/2015 05:14 PM, Doug Hellmann wrote:
> > Excerpts from Dmitry Tantsur's message of 2015-11-30 10:06:25 +0100:
> >> On 11/28/2015 02:48 PM, Doug Hellmann wrote:
> >>> Excerpts from Doug Hellmann's message of 2015-11-27 10:21:36 -0500:
>  Liaisons,
> 
>  We're making good progress on adding reno to service projects as
>  we head to the Mitaka-1 milestone. Thank you!
> 
>  We also need to add reno to all of the other deliverables with
>  changes that might affect deployers. That means clients and other
>  libraries, SDKs, etc. with configuration options or where releases
>  can change deployment behavior in some way. Now that most teams
>  have been through this conversion once, it should be easy to replicate
>  for the other repositories in a similar way.
> 
>  Libraries have 2 audiences for release notes: developers consuming
>  the library and deployers pushing out new versions of the libraries.
>  To separate the notes for the two audiences, and avoid doing manually
>  something that we have been doing automatically, we can use reno
>  just for deployer release notes (changes in support for options,
>  drivers, etc.). That means the library repositories that need reno
>  should have it configured just like for the service projects, with
>  the separate jobs and a publishing location different from their
>  existing developer documentation. The developer docs can continue
>  to include notes for the developer audience.
> >>>
> >>> I've had a couple of questions about this split for release notes. The
> >>> intent is for developer-focused notes to continue to come from commit
> >>> messages and in-tree documentation, while using reno for new and
> >>> additional deployer-focused communication. Most commits to libraries
> >>> won't need reno release notes.
> >>
> >> This looks like unnecessary overcomplication. Why not use such a
> >> convenient tool for both kinds of release notes instead of having us
> >> invent and maintain one more place to put release notes, now for
> >
> > In the past we have had rudimentary release notes and changelogs
> > for developers to read based on the git commit messages. Since
> > deployers and developers care about different things, we don't want
> > to make either group sift through the notes meant for the other.
> > So, we publish notes in different ways.
> 
> Hmm, so maybe for small libraries with few changes it's still fine to 
> publish them together, what do you think?

I'm not sure why you would want to do that. Publishing the ChangeLog
contents in the developer documentation is (or can be) completely
automatic. It should only be possible to add reno notes for
deployer-facing changes, and those notes will need to be written
in a way the deployer can understand, which is not necessarily a
requirement for a commit message.

> > The thing that is new here is publishing release notes for changes
> > in libraries that deployers need to know about. While the Oslo code
> > was in the incubator, and being copied into applications, it was
> > possible to detect deployer-focused changes like new or deprecated
> > configuration options in the application and put the notes there.
> > Using shared libraries means those changes can happen without
> > application developers being aware of them, so the library maintainers
> > need to be publishing notes. Using reno for those notes is consistent
> > with the way they are handled in the applications, so we're extending
> > one tool to more repositories.
> >
> >> developers? It's already not so easy to explain reno to newcomers, this
> >> idea makes it even harder...
> >
> > Can you tell me more about the difficulty you've had? I would like to
> > improve the documentation for reno and for how we use it.
> 
> Usually people are stuck at the "how do I do this at all" stage :) we've 
> even added it to the ironic developer FAQ. As to me, the official reno 
> documentation is nice enough (but see below), maybe people are not aware 
> of it.
> 
> Another "issue" (at least for our newcomers) with reno docs is that 
> http://docs.openstack.org/developer/reno/usage.html#generating-a-report 
> mentions the "reno report" command which is not something we all 
> actually use, we use these "tox -ereleasenotes" command. What is worse, 
> this command (I guess it's by design) does not catch release note files 
> that are just created locally. It took me time to figure out that I have 
> to commit release notes before "tox -ereleasenotes" would show them in 
> the rendered HTML.

The reno documentation is written for any user, not just OpenStack
developers. Those instructions should work if reno is installed,
even though we've wrapped reno in tox to make it simpler to run in
the gate. We can add some information about using tox to build
locally to the project team guide.

I'll look into 

[openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Joshua Harlow

Hi all,

I just wanted to bring up an issue, possible solution and get feedback 
on it from folks because it seems to be an on-going problem that shows 
up not when an application is initially deployed but as on-going 
operation and running of that application proceeds (ie after running for 
a period of time).


The jist of the problem is the following:

A <> has a need to ensure that no 
application on the same machine can manipulate a given resource on that 
same machine, so it uses the lock file pattern (acquire a *local* lock 
file for that resource, manipulate that resource, release that lock 
file) to do actions on that resource in a safe manner (note this does 
not ensure safety outside of that machine, lock files are *not* 
distributed locks).


The api that we expose from oslo is typically accessed via the following:

  oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None, 
external=False, lock_path=None, semaphores=None, delay=0.01)


or via its underlying library (that I extracted from oslo.concurrency 
and have improved to add more usefulness) @ 
http://fasteners.readthedocs.org/


The issue though for <> is that each of 
these projects now typically has a large amount of lock files that exist 
or have existed and no easy way to determine when those lock files can 
be deleted (afaik no? periodic task exists in said projects to clean up 
lock files, or to delete them when they are no longer in use...) so what 
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387 
appear and there is no a simple solution to clean lock files up (since 
oslo.concurrency is really not the right layer to know when a lock can 
or can not be deleted, only the application knows that...)


So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in > cleaning up these files (are they at all?)


Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single* 
lock file per project and use offsets into it as the way to lock. For 
example nova could/would create a 1MB (or larger/smaller) *empty* file 
for locks, that would allow for 1,048,576 locks to be used at the same 
time, which honestly should be way more than enough, and then there 
would not need to be any lock cleanup at all... Is there any reason this 
wasn't initially done back way when this lock file code was created? 
(https://github.com/harlowja/fasteners/pull/10 adds this functionality 
to the underlying library if people want to look it over)


In general would like to hear peoples thoughts/ideas/complaints/other,

-Josh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] Nova mid cycle details

2015-11-30 Thread Murray, Paul (HP Cloud)
The rates are listed on the hotel information page on that site. All include 
tax and breakfast.

Paul

From: Michael Still [mailto:mi...@stillhq.com]
Sent: 28 November 2015 04:39
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] Nova mid cycle details

Hey,

I filled in the first part of that page, but when it got to hotels I got 
confused. The web site doesn't seem to mention the night rate for the HP price. 
Do you know what that is?

Thanks,
Michael

On Sat, Nov 28, 2015 at 3:06 AM, Murray, Paul (HP Cloud) 
> wrote:
The Nova Mitaka mid cycle meetup is in Bristol, UK at the Hewlett Packard 
Enterprise office.

The mid cycle wiki page is here:
https://wiki.openstack.org/wiki/Sprints/NovaMitakaSprint

Note that there is a web site for signing up for the event and booking hotel 
rooms at a reduced event rate here:
https://starcite.smarteventscloud.com/hpe/NovaMidCycleMeeting

If you want to book a room at the event rate you do need to register on that 
site.

There is also an Eventbrite event that was created before the above web site 
was available. Do not worry if you have registered using Eventbrite, we will 
recognize those registrations as well. But if you do want to book a room you 
will need to register again on the above site.

Paul

Paul Murray
Nova Technical Lead, HPE Cloud
Hewlett Packard Enterprise
+44 117 316 2527


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Rackspace Australia
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Infra] Meeting Tuesday December 1st at 19:00 UTC

2015-11-30 Thread Elizabeth K. Joseph
Hi everyone,

The OpenStack Infrastructure (Infra) team is having our next weekly
meeting on Tuesday December 1st, at 19:00 UTC in #openstack-meeting

Meeting agenda available here:
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting

Anyone is welcome to to add agenda items and everyone interested in
the project infrastructure and process surrounding automated testing
and deployment is encouraged to attend.

In case you missed it or would like a refresher, the meeting minutes
and full logs from our last meeting are available:

Minutes: 
http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-11-24-19.01.html
Minutes (text):
http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-11-24-19.01.txt
Log: 
http://eavesdrop.openstack.org/meetings/infra/2015/infra.2015-11-24-19.01.log.html

-- 
Elizabeth Krumbach Joseph || Lyz || pleia2

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Performance] Filling performance team working items list and taking part in their resolving

2015-11-30 Thread Dina Belova
Matt,

I guess it's good solution. Thanks for doing that!

Cheers,
Dina

On Mon, Nov 30, 2015 at 6:30 PM, Matt Riedemann 
wrote:

>
>
> On 11/27/2015 3:54 AM, Dina Belova wrote:
>
>> Hey OpenStack devs and operators!
>>
>> Folks, I would like to share list of working items Performance Team is
>> currently having in the backlog -
>> https://etherpad.openstack.org/p/perf-zoom-zoom [Work Items to grab]
>> section. I'm really encouraging you to fill it with concrete pieces of
>> work you think will be useful and take part in the
>> development/investigation by assigning some of them to yourself and
>> working on them :)
>>
>> Cheers,
>> Dina
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> One of the work items was to enable the nova metadata service in a large
> ops job. There are two voting large ops jobs in nova, one runs with
> nova-network and the other runs with neutron. Besides those differences,
> I'm not sure if there is any other difference in the jobs. So I guess we'd
> just need to pick which one runs the nova-api-metadata service rather than
> using config drive. The only other job I know of that runs that is the
> postgres job and that runs nova-network, so I'd say we turn on n-api-meta
> in the neutron large ops job.
>
> Are there any issues with that?
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

Best regards,

Dina Belova

Software Engineer

Mirantis Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Call for review focus

2015-11-30 Thread Carl Baldwin
++

On Wed, Nov 25, 2015 at 3:05 PM, Assaf Muller  wrote:
> On Mon, Nov 23, 2015 at 7:02 AM, Rossella Sblendido  
> wrote:
>>
>>
>> On 11/20/2015 03:54 AM, Armando M. wrote:
>>>
>>>
>>>
>>> On 19 November 2015 at 18:26, Assaf Muller >> > wrote:
>>>
>>> On Wed, Nov 18, 2015 at 9:14 PM, Armando M. >> > wrote:
>>> > Hi Neutrites,
>>> >
>>> > We are nearly two weeks away from the end of Mitaka 1.
>>> >
>>> > I am writing this email to invite you to be mindful to what you
>>> review,
>>> > especially in the next couple of weeks. Whenever you have the time
>>> to review
>>> > code, please consider giving priority to the following:
>>> >
>>> > Patches that target blueprints targeted for Mitaka;
>>> > Patches that target bugs that are either critical or high;
>>> > Patches that target rfe-approved 'bugs';
>>> > Patches that target specs that have followed the most current
>>> submission
>>> > process;
>>>
>>> Is it possible to create Gerrit dashboards for patches that answer
>>> these
>>> criteria, and then persist the links in Neutron's dashboards devref
>>> page?
>>> http://docs.openstack.org/developer/neutron/dashboards/index.html
>>> That'd be super useful.
>>>
>>>
>>> We should look into that, but to be perfectly honest I am not sure how
>>> easy it would be, since we'd need to cross-reference content that lives
>>> into gerrit as well as launchpad. Would that even be possible?
>>
>>
>> To cross-reference we can use the bug ID or the blueprint name.
>>
>> I created a script that queries launchpad to get:
>> 1) Bug number of the bugs tagged with approved-rfe
>> 2) Bug number of the critical/high bugs
>> 3) list of blueprints targeted for the current milestone (mitaka-1)
>>
>> With this info the script builds a .dash file that can be used by
>> gerrit-dash-creator [2] to produce a dashboard url .
>>
>> The script prints also the queries that can be used in gerrit UI directly,
>> e.g.:
>> Critical/High Bugs
>> (topic:bug/1399249 OR topic:bug/1399280 OR topic:bug/1443421 OR
>> topic:bug/1453350 OR topic:bug/1462154 OR topic:bug/1478100 OR
>> topic:bug/1490051 OR topic:bug/1491131 OR topic:bug/1498790 OR
>> topic:bug/1505575 OR topic:bug/1505843 OR topic:bug/1513678 OR
>> topic:bug/1513765 OR topic:bug/1514810)
>>
>>
>> This is the dashboard I get right now [3]
>>
>> I tried in many ways to get Gerrit to filter patches if the commit message
>> contains a bug ID. Something like:
>>
>> (message:"#1399249" OR message:"#1399280" OR message:"#1443421" OR
>> message:"#1453350" OR message:"#1462154" OR message:"#1478100" OR
>> message:"#1490051" OR message:"#1491131" OR message:"#1498790" OR
>> message:"#1505575" OR message:"#1505843" OR message:"#1513678" OR
>> message:"#1513765" OR message:"#1514810")
>>
>> but it doesn't work well, the result of the filter contains patches that
>> have nothing to do with the bugs queried.
>> That's why I had to filter using the topic.
>>
>> CAVEAT: To make the dashboard work, bug fixes must use the topic "bug/ID"
>> and patches implementing a blueprint the topic "bp/name". If a patch is not
>> following this convention it won't be showed in the dashboard, since the
>> topic is used as filter. Most of us use this convention already anyway so I
>> hope it's not too much of a burden.
>>
>> Feedback is appreciated :)
>
> Rossella this is exactly what I wanted :) Let's iterate on the patch
> and merge it.
> We could then consider running the script automatically on a daily
> basis and publishing the
> resulting URL in a nice bookmarkable place.
>
>>
>> [1] https://review.openstack.org/248645
>> [2] https://github.com/openstack/gerrit-dash-creator
>> [3] https://goo.gl/sglSbp
>>
>>>
>>> Btw, I was looking at the current blueprint assignments [1] for Mitaka:
>>> there are some blueprints that still need assignee, approver and
>>> drafter; we should close the gap. If there are volunteers, please reach
>>> out to me.
>>>
>>> Thanks,
>>> Armando
>>>
>>> [1] https://blueprints.launchpad.net/neutron/mitaka/+assignments
>>>
>>>
>>> >
>>> > Everything else should come later, no matter how easy or interesting
>>> it is
>>> > to review; remember that as a community we have the collective duty
>>> to work
>>> > towards a common (set of) target(s), as being planned in
>>> collaboration with
>>> > the Neutron Drivers team and the larger core team.
>>> >
>>> > I would invite submitters to ensure that the Launchpad resources
>>> > (blueprints, and bug report) capture the most updated view in terms
>>> of
>>> > patches etc. Work with your approver to help him/her be focussed
>>> where it
>>> > matters most.
>>> >
>>> > Finally, we had plenty of discussions at the design summit, and some
>>> of
>>> > those discussions will have to be followed 

Re: [openstack-dev] [all][glance] Add Sabari Kumar Murugesan <smuruge...@vmware.com>

2015-11-30 Thread Ian Cordasco
On 11/23/15, 14:20, "Flavio Percoco"  wrote:

>Greetings,
>
>I'd like to propose adding Sabari Kumar Murugesan to the glance-core
>team. Sabari has been contributing for quite a bit to the project with
>great reviews and he's also been providing great feedback in matters
>related to the design of the service, libraries and other areas of the
>team.
>
>I believe he'd be a great addition to the glance-core team as he has
>demonstrated a good knowledge of the code, service and project's
>priorities.
>
>If Sabari accepts to join and there are no objections from other
>members of the community, I'll proceed to add Sabari to the team in a
>week from now.
>
>Thanks,
>Flavio
>
>-- 
>@flaper87
>Flavio Percoco

I'm no longer a core, but I'm +1 on this.

Congrats Sabari!

--
Ian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Call for review focus

2015-11-30 Thread Carl Baldwin
On Tue, Nov 24, 2015 at 4:47 AM, Rossella Sblendido  wrote:
>> I looked for the address scopes blueprint [1] which is targeted for
>> Mitaka-1 [2] and there are 6 (or 5, one is in the gate) patches on the
>> bp/address-scopes topic [3].  It isn't obvious to me yet why it didn't
>> get picked up on the dashboard.  I've only started to look in to this
>> and may not have much time right now.  I wondered if you could easily
>> tell why it didn't get picked up.
>>
>> Isn't it missing bp/ ? From the URL I can only see topic:address-scope,
>> which isn't the right one.
>
>
> Yes Armando is right. I fixed that. Another reason is that I am filtering
> out patches that are WIP or that failed Jenkins tests. This can be changed
> anyway. This is what I get now (after fixing the missing 'bp/') [1]

I tend to think that eliminating patches that are failing tests may
not be a good thing to do in general.  I think it makes the dashboard
too dynamic in the face of all of the unreliable tests that we see
come and go.  It would go dark when we see those check queue bombs
that fail all of the patches.  This is just my opinion and I could
probably adjust the end result to suit my tastes.

I do filter patch sets with failing tests in some of my own personal queries.

Carl

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

2015-11-30 Thread Dmitry Tantsur

On 11/30/2015 06:24 PM, Anita Kuno wrote:

On 11/30/2015 12:17 PM, Dmitry Tantsur wrote:

On 11/30/2015 05:34 PM, Anita Kuno wrote:

On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:

Hi,



   This is to announce that  we have  setup  a  Third Party CI
environment
for Proliant iLO Drivers. The results will be posted  under "HP
Proliant CI
check" section in Non-voting mode.   We will be  running the basic
deploy
tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We
will
first  pursue to make the results consistent and over a period of
time we
will try to promote it to voting mode.



 For more information check the Wiki:
https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci
,
for any issues please contact ilo_driv...@groups.ext.hpe.com





Thanks & Regards,

Gururaja Grandhi

R Project Manager

HPE Proliant  Ironic  Project



__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



Please do not post announcements to the mailing list about the existence
of your third party ci system.


Could you please explain why? As a developer I appreciated this post.



Ensure your third party ci system is listed here:
https://wiki.openstack.org/wiki/ThirdPartySystems (there are
instructions on the bottom of the page) as well as fill out a template
on your system so that folks can find your third party ci system the
same as all other third party ci systems.


Wiki is not an announcement FWIW.


If Ironic wants to hear about announced drivers they have agreed to do
so as part of their weekly irc meeting:
2015-11-30T17:19:55   I think it is reasonable for each
driver team, if they want to announce it in the meeting, to do so on the
whiteboard section for their driver. we'll all see that in the weekly
meeting
2015-11-30T17:20:08   but it will avoid spamming the whole
openstack list

http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2015-11-30.log


I was there and I already said that I'm not buying into "spamming the 
list" argument. There are much less important things that I see here 
right now, even though I do actively use filters to only see potentially 
relevant things. We've been actively (and not very successfully) 
encouraging people to use ML instead of IRC conversations (or even 
private messages and video chats), and this thread does not seem in line 
with it.




Thank you,
Anita.





Ensure you are familiar with the requirements for third party systems
listed here:
http://docs.openstack.org/infra/system-config/third_party.html#requirements


Thank you,
Anita.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [ironic] Announcing Third Party CI for Proliant iLO Drivers

2015-11-30 Thread Anita Kuno
On 11/30/2015 12:33 PM, Dmitry Tantsur wrote:
> On 11/30/2015 06:24 PM, Anita Kuno wrote:
>> On 11/30/2015 12:17 PM, Dmitry Tantsur wrote:
>>> On 11/30/2015 05:34 PM, Anita Kuno wrote:
 On 11/30/2015 11:25 AM, Gururaj Grandhi wrote:
> Hi,
>
>
>
>This is to announce that  we have  setup  a  Third Party CI
> environment
> for Proliant iLO Drivers. The results will be posted  under "HP
> Proliant CI
> check" section in Non-voting mode.   We will be  running the basic
> deploy
> tests for  iscsi_ilo and agent_ilo drivers  for the check queue.  We
> will
> first  pursue to make the results consistent and over a period of
> time we
> will try to promote it to voting mode.
>
>
>
>  For more information check the Wiki:
> https://wiki.openstack.org/wiki/Ironic/Drivers/iLODrivers/third-party-ci
>
> ,
> for any issues please contact ilo_driv...@groups.ext.hpe.com
>
>
>
>
>
> Thanks & Regards,
>
> Gururaja Grandhi
>
> R Project Manager
>
> HPE Proliant  Ironic  Project
>
>
>
> __
>
>
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

 Please do not post announcements to the mailing list about the
 existence
 of your third party ci system.
>>>
>>> Could you please explain why? As a developer I appreciated this post.
>>>

 Ensure your third party ci system is listed here:
 https://wiki.openstack.org/wiki/ThirdPartySystems (there are
 instructions on the bottom of the page) as well as fill out a template
 on your system so that folks can find your third party ci system the
 same as all other third party ci systems.
>>>
>>> Wiki is not an announcement FWIW.
>>
>> If Ironic wants to hear about announced drivers they have agreed to do
>> so as part of their weekly irc meeting:
>> 2015-11-30T17:19:55   I think it is reasonable for each
>> driver team, if they want to announce it in the meeting, to do so on the
>> whiteboard section for their driver. we'll all see that in the weekly
>> meeting
>> 2015-11-30T17:20:08   but it will avoid spamming the whole
>> openstack list
>>
>> http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-3/%23openstack-meeting-3.2015-11-30.log
>>
> 
> I was there and I already said that I'm not buying into "spamming the
> list" argument. There are much less important things that I see here
> right now, even though I do actively use filters to only see potentially
> relevant things. We've been actively (and not very successfully)
> encouraging people to use ML instead of IRC conversations (or even
> private messages and video chats), and this thread does not seem in line
> with it.

Please discuss this with the leadership of your project.

All announcements about the existence of a third party ci will be
redirected to the third party systems wikipage.

Thank you,
Anita.

> 
>>
>> Thank you,
>> Anita.
>>
>>>

 Ensure you are familiar with the requirements for third party systems
 listed here:
 http://docs.openstack.org/infra/system-config/third_party.html#requirements



 Thank you,
 Anita.
>>>
>>
>> __
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Ruby Loo
On 30 November 2015 at 10:19, Derek Higgins  wrote:

> Hi All,
>
> A few months tripleo switch from its devtest based CI to one that was
> based on instack. Before doing this we anticipated disruption in the ci
> jobs and removed them from non tripleo projects.
>
> We'd like to investigate adding it back to heat and ironic as these
> are the two projects where we find our ci provides the most value. But we
> can only do this if the results from the job are treated as voting.
>

What does this mean? That the tripleo job could vote and do a -1 and block
ironic's gate?


>
> In the past most of the non tripleo projects tended to ignore the
> results from the tripleo job as it wasn't unusual for the job to broken for
> days at a time. The thing is, ignoring the results of the job is the reason
> (the majority of the time) it was broken in the first place.
> To decrease the number of breakages we are now no longer running
> master code for everything (for the non tripleo projects we bump the
> versions we use periodically if they are working). I believe with this
> model the CI jobs we run have become a lot more reliable, there are still
> breakages but far less frequently.
>
> What I proposing is we add at least one of our tripleo jobs back to both
> heat and ironic (and other projects associated with them e.g. clients,
> ironicinspector etc..), tripleo will switch to running latest master of
> those repositories and the cores approving on those projects should wait
> for a passing CI jobs before hitting approve. So how do people feel about
> doing this? can we give it a go? A couple of people have already expressed
> an interest in doing this but I'd like to make sure were all in agreement
> before switching it on.
>
> This seems to indicate that the tripleo jobs are non-voting, or at least
won't block the gate -- so I'm fine with adding tripleo jobs to ironic. But
if you want cores to wait/make sure they pass, then shouldn't they be
voting? (Guess I'm a bit confused.)

--ruby
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Sahara] Debugging via PyCharm (pydevd)

2015-11-30 Thread Chad Roberts
I submitted a bug (https://bugs.launchpad.net/sahara/+bug/1521266) and have
asked in-channel, but I'll also post here just in case it rings a bell for
someone.

Recently, Sahara was changed to use oslo.service to launch our wsgi api
server.  Prior to that change, I was able to successfully run and debug the
sahara-api process using PyCharm.

After the change to use oslo.service as the launcher, I am still able to
run sahara-api using the same config that has worked for as long as I can
remember, but when I run in debug mode, the api appears to start up
normally, but any request I send never receives a response and any
breakpoints I've tried never seem to get hit.

If I backout the changes to use oslo.service, I am able to debug
successfully again.

Any chance that sort of thing sounds familiar to anyone?

Thanks,
Chad
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [keystone][all] keystone-spec proposal freeze date and using roll call instead of +2/+2/+A for specs

2015-11-30 Thread Steve Martinelli


As a reminder: keystone spec freeze date is the end of mitaka-1 (this
friday), any spec being proposed for mitaka will have to go through an
exception process next week.

For the next keystone meeting (tomorrow) I'd like to propose we use a roll
call / vote mechanism on each open specs that is proposed against the
mitaka release. If a majority of the cores (50% or more) agree that a spec
aligns with project plans for mitaka, then we should get it merged by
friday (clean up any wording and proposed APIs) so the author can start
working on it on monday.

I added [all] because I wanted to know how other projects approach this
subject. I think a roll call feature for -specs would be great, similar to
how there is one for governance changes. Do others just use +2/+2/+A? In
keystone we've often let specs sit for a little to give other cores a
chance to chime in. Thoughts?

Thanks,

Steve Martinelli
OpenStack Keystone Project Team Lead
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues

2015-11-30 Thread Paul Michali
Check out https://freenode.net/irc_servers.shtml which lists the servers. I
was using irc.freenode.net. Switched to weber.freenode.net and able to
connect.

(now everyone will hop on that one and I'll have to pick another :)



On Mon, Nov 30, 2015 at 2:46 PM Clark, Jay  wrote:

> Can't connect either. Dead in the water
>
> Regards,
> Jay Clark
> Sr. OpenStack Deployment Engineer
> E: jason.t.cl...@hpe.com
> H: 919.341.4670
> M: 919.345.1127
> IRC (freenode): jasondotstar
>
> 
> From: lichen.hangzhou [lichen.hangz...@gmail.com]
> Sent: Monday, November 30, 2015 9:17 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: openstack-in...@lists.openstack.org
> Subject: Re: [openstack-dev] [OpenStack-Infra] IRC Bot issues
>
> Can't connect +1 and web client do not work  :(
>
> -chen
>
> At 2015-11-30 22:08:12, "Hinds, Luke (Nokia - GB/Bristol)" <
> luke.hi...@nokia.com> wrote:
> Me too.  It is possible to get on using the web client though;
> https://webchat.freenode.net/ .
>
> On Mon, 2015-11-30 at 14:00 +, EXT Dugger, Donald D wrote:
> I can’t even connect to the IRC server at all, can others get it?
>
> --
> Don Dugger
> "Censeo Toto nos in Kansa esse decisse." - D. Gale
> Ph: 303/443-3786
>
> From: Joshua Hesketh [mailto:joshua.hesk...@gmail.com joshua.hesk...@gmail.com>]
> Sent: Monday, November 30, 2015 2:50 AM
> To: OpenStack Development Mailing List (not for usage questions) <
> openstack-dev@lists.openstack.org>;
> openstack-infra >
> Subject: [openstack-dev] IRC Bot issues
>
> Hi all,
> Freenode are currently experiencing a severe DDoS attack that are having
> an effect on our bots. As such the meetbot, irc logging and gerrit watcher
> are interminably available.
>
> We expect the bots to resume their normal function once Freenode has
> recovered. For now, meetings may have to be postponed or minuted by hand.
> Cheers,
> Josh
>
> ___
> OpenStack-Infra mailing list
> openstack-in...@lists.openstack.org openstack-in...@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
>
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [puppet] [ceph] Puppet Ceph CI

2015-11-30 Thread David Moreau Simard
Hey Adam,

A bit late here, sorry.
Ceph works fine with OpenStack Kilo but at the time we developed the
integration tests for puppet-ceph with Kilo, there were some issues
specific to our test implementation and we chose to settle with Juno
at the time.

On the topic of CI, I can no longer sponsor the third party CI
(through my former employer, iWeb) as I am with Red Hat now.
I see this as an opportunity to drop the custom system tests with
vagrant and instead improve the acceptance tests.

What do you think ?


David Moreau Simard
Senior Software Engineer | Openstack RDO

dmsimard = [irc, github, twitter]


On Mon, Nov 23, 2015 at 6:45 PM, Adam Lawson  wrote:
> I'm confused, what is the context here? We use Ceph with OpenStack Kilo
> without issue.
>
> On Nov 23, 2015 2:28 PM, "David Moreau Simard"  wrote:
>>
>> Last I remember, David Gurtner tried to use Kilo instead of Juno but
>> he bumped into some problems and we settled for Juno at the time [1].
>> At this point we should already be testing against both Liberty and
>> Infernalis, we're overdue for an upgrade in that regard.
>>
>> But, yes, +1 to split acceptance tests:
>> 1) Ceph
>> 2) Ceph + Openstack
>>
>> Actually learning what failed is indeed challenging sometimes, I don't
>> have enough experience with the acceptance testing to suggest anything
>> better.
>> We have the flexibility of creating different logfiles, maybe we can
>> find a way to split out the relevant bits into another file.
>>
>> [1]: https://review.openstack.org/#/c/153783/
>>
>> David Moreau Simard
>> Senior Software Engineer | Openstack RDO
>>
>> dmsimard = [irc, github, twitter]
>>
>>
>> On Mon, Nov 23, 2015 at 2:45 PM, Andrew Woodward  wrote:
>> > I think I have a good lead on the recent failures in openstack / swift /
>> > radosgw integration component that we have since disabled. It looks like
>> > there is a oslo.config version upgrade conflict in the Juno repo we
>> > where
>> > using for CentOS. I think moving to Kilo will help sort this out, but at
>> > the
>> > same time I think it would be prudent to separate the Ceph v.s.
>> > OpenStack
>> > integration into separate jobs so that we have a better idea of which is
>> > a
>> > problem. If there is census for this, I'd need some direction / help, as
>> > well as set them up as non-voting for now.
>> >
>> > Looking into this I also found that the only place that we do
>> > integration
>> > any of the cephx logic was in the same test so we will need to create a
>> > component for it in the ceph integration as well as use it in the
>> > OpenStack
>> > side.
>> >
>> > Lastly un-winding the integration failure seemed overly complex. Is
>> > there a
>> > way that we can correlate the test status inside the job at a high level
>> > besides the entire job passed / failed without breaking them into
>> > separate
>> > jobs?
>> > --
>> >
>> > --
>> >
>> > Andrew Woodward
>> >
>> > Mirantis
>> >
>> > Fuel Community Ambassador
>> >
>> > Ceph Community
>> >
>> >
>> >
>> > __
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Robert Collins
On 1 December 2015 at 08:37, Ben Nemec  wrote:
> On 11/30/2015 12:42 PM, Joshua Harlow wrote:
>> Hi all,
>>
>> I just wanted to bring up an issue, possible solution and get feedback
>> on it from folks because it seems to be an on-going problem that shows
>> up not when an application is initially deployed but as on-going
>> operation and running of that application proceeds (ie after running for
>> a period of time).
>>
>> The jist of the problem is the following:
>>
>> A <> has a need to ensure that no
>> application on the same machine can manipulate a given resource on that
>> same machine, so it uses the lock file pattern (acquire a *local* lock
>> file for that resource, manipulate that resource, release that lock
>> file) to do actions on that resource in a safe manner (note this does
>> not ensure safety outside of that machine, lock files are *not*
>> distributed locks).
>>
>> The api that we expose from oslo is typically accessed via the following:
>>
>>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
>> external=False, lock_path=None, semaphores=None, delay=0.01)
>>
>> or via its underlying library (that I extracted from oslo.concurrency
>> and have improved to add more usefulness) @
>> http://fasteners.readthedocs.org/
>>
>> The issue though for <> is that each of
>> these projects now typically has a large amount of lock files that exist
>> or have existed and no easy way to determine when those lock files can
>> be deleted (afaik no? periodic task exists in said projects to clean up
>> lock files, or to delete them when they are no longer in use...) so what
>> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
>> appear and there is no a simple solution to clean lock files up (since
>> oslo.concurrency is really not the right layer to know when a lock can
>> or can not be deleted, only the application knows that...)
>>
>> So then we get a few creative solutions like the following:
>>
>> - https://review.openstack.org/#/c/241663/
>> - https://review.openstack.org/#/c/239678/
>> - (and others?)
>>
>> So I wanted to ask the question, how are people involved in <> favorite openstack project>> cleaning up these files (are they at all?)
>>
>> Another idea that I have been proposing also is to use offset locks.
>>
>> This would allow for not creating X lock files, but create a *single*
>> lock file per project and use offsets into it as the way to lock. For
>> example nova could/would create a 1MB (or larger/smaller) *empty* file
>> for locks, that would allow for 1,048,576 locks to be used at the same
>> time, which honestly should be way more than enough, and then there
>> would not need to be any lock cleanup at all... Is there any reason this
>> wasn't initially done back way when this lock file code was created?
>> (https://github.com/harlowja/fasteners/pull/10 adds this functionality
>> to the underlying library if people want to look it over)
>
> I think the main reason was that even with a million locks available,
> you'd have to find a way to hash the lock names to offsets in the file,
> and a million isn't a very large collision space for that.  Having two
> differently named locks that hashed to the same offset would lead to
> incredibly confusing bugs.
>
> We could switch to requiring the projects to provide the offsets instead
> of hashing a string value, but that's just pushing the collision problem
> off onto every project that uses us.
>
> So that's the problem as I understand it, but where does that leave us
> for solutions?  First, there's
> https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
> which allows consumers to delete lock files when they're done with them.
>  Of course, in that case the onus is on the caller to make sure the lock
> couldn't possibly be in use anymore.
>
> Second, is this actually a problem?  Modern filesystems have absurdly
> large limits on the number of files in a directory, so it's highly
> unlikely we would ever exhaust that, and we're creating all zero byte
> files so there shouldn't be a significant space impact either.  In the
> past I believe our recommendation has been to simply create a cleanup
> job that runs on boot, before any of the OpenStack services start, that
> deletes all of the lock files.  At that point you know it's safe to
> delete them, and it prevents your lock file directory from growing forever.

Not that high - ext3 (still the default for nova ephemeral
partitions!) has a limit of 64k in one directory.

That said, I don't disagree - my thinkis is that we should advise
putting such files on a tmpfs.

-Rob

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [ironic] weekly subteam status report

2015-11-30 Thread Ruby Loo
Hi,


We are elated to present this week's subteam report for Ironic. As usual,
this is pulled directly from the Ironic whiteboard[0] and formatted.


Bugs (dtantsur)

===

(diff with Nov 23)

- Open: 174 (+3). 11 new, 60 in progress, 0 critical, 16 high and 11
incomplete

- Nova bugs with Ironic tag: 24. 0 new, 0 critical, 0 high

- Inspector bugs: 13 (-1). 0 new, 0 critical, 6 high

- bug number is slowly growing since 28.09.2015 (135 open bugs)

- bugs to be aware of before the release this week:

- https://bugs.launchpad.net/ironic/+bug/1507738 - gives headache to people
with CentOS 7, might be considered a regression (status: fix on review)

- https://bugs.launchpad.net/ironic/+bug/1512544 - "grenade jobs are
failing" does not sound promising (status: in progress, no patch attached)

- https://bugs.launchpad.net/ironic/+bug/1408067 - do we still experience
these?



Network isolation (Neutron/Ironic work) (jroll)

===

- nova spec has landed; 2/3 patches are updated and passing unit tests

- ironic reviews still ongoing...



Live upgrades (lucasagomes, lintan)

===

- Submit a patch to add test to enforce object version bump correctly

- https://review.openstack.org/#/c/249624/ MERGED



Boot interface refactor (jroll)

===

- done!

- let's remove this one for next meeting \o/



Parallel tasks with futurist (dtantsur)

===

- still refactoring manager.py a bit:
https://review.openstack.org/#/c/249938/ is WIP



Node filter API and claims endpoint (jroll, devananda)

==

- spec still in review - https://review.openstack.org/#/c/204641



Multiple compute hosts (jroll, devananda)

=

- dependant on node filter API


ironic-lib adoption (rloo)

==

- ironic-lib 0.4.0 has been released and updated in requirements, patch is
almost ready (needs reno) but holding off til after ironic release:
https://review.openstack.org/#/c/184443/



Nova Liaisons (jlvillal & mrda)

===

- No meetiing/updates



Testing/Quality (jlvillal/lekha/krtaylor)

=

- No meeting/updates



Inspector (dtansur)

===

- Fully switched to Reno for both inspector projects

- e.g
http://docs.openstack.org/releasenotes/python-ironic-inspector-client/unreleased.html

- Documentation is now using Sphinx, will appear on
docs.openstack.org/developer/ironic-inspector this week

- python-ironic-inspector-client to be released this week, ironic-inspector
does not seem to have enough changes

- ironic-inspector-specs addition is proposed to the TC



Bifrost (TheJulia)

==

- Gate job is working again, presently looking at refactoring non-voting
test job.



webclient (krotscheck / betherly)

=

- Panel currently being wrapped in plugin. Actions on the nodes are
working. Details page to do then start port



Drivers:



DRAC (ifarkas/lucas)



- patches refactoring the driver to use python-dracclient instead of
pywsman are on gerrit


iLO (wanyen)



- 3rd party CI:
http://lists.openstack.org/pipermail/openstack-dev/2015-November/080806.html


iRMC (naohirot)

---

https://review.openstack.org//#/q/owner:+naohirot+status:+open,n,z

- Status: Reactive (solicited for core team's review)

- iRMC out of band inspection (bp/ironic-node-properties-discovery)

- Status: Active

- Enhance Power Interface for Soft Reboot and NMI
(bp/enhance-power-interface-for-soft-reboot-and-nmi)

- Add 'abort' support for Soft Power Off and Inject NMI
(bp/task-control-functions-for-long-running-tasks)

- iRMC OOB rescue mode support (bp/irmc-oob-rescue-mode-support)

- Status: Done, thanks!

- New boot driver interface for iRMC drivers (bp/new-boot-interface)


.


Until next week,

--ruby


[0] https://etherpad.openstack.org/p/IronicWhiteBoard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Clint Byrum
Excerpts from Joshua Harlow's message of 2015-11-30 10:42:53 -0800:
> Hi all,
> 
> I just wanted to bring up an issue, possible solution and get feedback 
> on it from folks because it seems to be an on-going problem that shows 
> up not when an application is initially deployed but as on-going 
> operation and running of that application proceeds (ie after running for 
> a period of time).
> 
> The jist of the problem is the following:
> 
> A <> has a need to ensure that no 
> application on the same machine can manipulate a given resource on that 
> same machine, so it uses the lock file pattern (acquire a *local* lock 
> file for that resource, manipulate that resource, release that lock 
> file) to do actions on that resource in a safe manner (note this does 
> not ensure safety outside of that machine, lock files are *not* 
> distributed locks).
> 
> The api that we expose from oslo is typically accessed via the following:
> 
>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None, 
> external=False, lock_path=None, semaphores=None, delay=0.01)
> 
> or via its underlying library (that I extracted from oslo.concurrency 
> and have improved to add more usefulness) @ 
> http://fasteners.readthedocs.org/
> 
> The issue though for <> is that each of 
> these projects now typically has a large amount of lock files that exist 
> or have existed and no easy way to determine when those lock files can 
> be deleted (afaik no? periodic task exists in said projects to clean up 
> lock files, or to delete them when they are no longer in use...) so what 
> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387 
> appear and there is no a simple solution to clean lock files up (since 
> oslo.concurrency is really not the right layer to know when a lock can 
> or can not be deleted, only the application knows that...)
> 
> So then we get a few creative solutions like the following:
> 
> - https://review.openstack.org/#/c/241663/
> - https://review.openstack.org/#/c/239678/
> - (and others?)
> 
> So I wanted to ask the question, how are people involved in < favorite openstack project>> cleaning up these files (are they at all?)
> 
> Another idea that I have been proposing also is to use offset locks.
> 
> This would allow for not creating X lock files, but create a *single* 
> lock file per project and use offsets into it as the way to lock. For 
> example nova could/would create a 1MB (or larger/smaller) *empty* file 
> for locks, that would allow for 1,048,576 locks to be used at the same 
> time, which honestly should be way more than enough, and then there 
> would not need to be any lock cleanup at all... Is there any reason this 
> wasn't initially done back way when this lock file code was created? 
> (https://github.com/harlowja/fasteners/pull/10 adds this functionality 
> to the underlying library if people want to look it over)

This is really complicated, and basically just makes the directory of
lock files _look_ clean. But it still leaves each offset stale, and has
to be cleaned anyway.

Fasteners already has process locks that use fcntl/flock.

These locks provide enough to allow you to infer things about.  the owner
of the lock file. If there's no process still holding the exclusive lock
when you try to lock it, then YOU own it, and thus control the resource.

A cron job which tries to flock anything older than ${REASONABLE_TIME}
and deletes them seems fine. Whatever process was trying to interact
with the resource is gone at that point.

Now, anything that needs to safely manage a resource beyond without a
live process will need to keep track of its own state and be idempotent
anyway. IMO this isn't something lock files alone solve well. I believe
you're familiar with a library named taskflow that is supposed to help
write code that does this better ;). Even without taskflow, if you are
trying to do something exclusive without a single process that stays
alive, you need to do _something_ to keep track of state and restart
or revert that flow. That is a state management problem, not a locking
problem.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] Versioned notifications... who cares about the version?

2015-11-30 Thread Andrew Laski

On 11/30/15 at 07:32am, Sean Dague wrote:

On 11/24/2015 10:09 AM, John Garbutt wrote:

On 24 November 2015 at 15:00, Balázs Gibizer
 wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 24, 2015 15:35
On 11/24/15 at 10:26am, Balázs Gibizer wrote:

From: Ryan Rossiter [mailto:rlros...@linux.vnet.ibm.com]
Sent: November 23, 2015 22:33
On 11/23/2015 2:23 PM, Andrew Laski wrote:

On 11/23/15 at 04:43pm, Balázs Gibizer wrote:

From: Andrew Laski [mailto:and...@lascii.com]
Sent: November 23, 2015 17:03

On 11/23/15 at 08:54am, Ryan Rossiter wrote:



On 11/23/2015 5:33 AM, John Garbutt wrote:

On 20 November 2015 at 09:37, Balázs Gibizer
 wrote:






There is a bit I am conflicted/worried about, and thats when we
start including verbatim, DB objects into the notifications. At
least you can now quickly detect if that blob is something
compatible with your current parsing code. My preference is
really to keep the Notifications as a totally separate object
tree, but I am sure there are many cases where that ends up
being seemingly stupid duplicate work. I am not expressing this
well in text form :(

Are you saying we don't want to be willy-nilly tossing DB
objects across the wire? Yeah that was part of the rug-pulling
of just having the payload contain an object. We're
automatically tossing everything with the object then, whether
or not some of that was supposed to be a secret. We could add
some sort of property to the field like
dont_put_me_on_the_wire=True (or I guess a
notification_ready() function that helps an object sanitize
itself?) that the notifications will look at to know if it puts
that on the wire-serialized dict, but that's adding a lot more
complexity and work to a pile that's already growing rapidly.


I don't want to be tossing db objects across the wire.  But I
also am not convinced that we should be tossing the current
objects over the wire either.
You make the point that there may be things in the object that
shouldn't be exposed, and I think object version bumps is another
thing to watch out for.
So far the only object that has been bumped is Instance but in
doing so no notifications needed to change.  I think if we just
put objects into notifications we're coupling the notification
versions to db or RPC changes unnecessarily.  Some times they'll
move together but other times, like moving flavor into
instance_extra, there's no reason to bump notifications.



Sanitizing existing versioned objects before putting them to the
wire is not hard to do.
You can see an example of doing it in
https://review.openstack.org/#/c/245678/8/nova/objects/service.py,
cm
L382.
We don't need extra effort to take care of minor version bumps
because that does not break a well written consumer. We do have to
take care of the major version bumps but that is a rare event and
therefore can be handled one by one in a way John suggested, by
keep sending the previous major version for a while too.


That review is doing much of what I was suggesting.  There is a
separate notification and payload object.  The issue I have is that
within the ServiceStatusPayload the raw Service object and version
is being dumped, with the filter you point out.  But I don't think
that consumers really care about tracking Service object versions
and dealing with compatibility there, it would be easier for them
to track the ServiceStatusPayload version which can remain
relatively stable even if Service is changing to adapt to db/RPC changes.

Not only do they not really care about tracking the Service object
versions, they probably also don't care about what's in that filter list.

But I think you're getting on the right track as to where this needs
to go. We can integrate the filtering into the versioning of the payload.
But instead of a blacklist, we turn the filter into a white list. If
the underlying object adds a new field that we don't want/care if
people know about, the payload version doesn't have to change. But if
we add something (or if we're changing the existing fields) that we
want to expose, we then assert that we need to update the version of
the payload, so the consumer can look at the payload and say "oh, in
1.x, now I get ___" and can add the appropriate checks/compat.
Granted with this you can get into rebase nightmares ([1] still
haunts me in my sleep), but I don't see us frantically changing the
exposed fields all too often. This way gives us some form of
pseudo-pinning of the subobject. Heck, in this method, we could even
pass the whitelist on the wire right? That way we tell the consumer

explicitly what's available to them (kinda like a fake schema).


I think see your point, and it seems like a good way forward. Let's
turn the black list to a white list. Now I'm thinking about creating a
new Field type something like WhiteListedObjectField which get a type
name (as the ObjectField) but also get a white_list that describes which

fields 

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Ben Nemec
On 11/30/2015 12:42 PM, Joshua Harlow wrote:
> Hi all,
> 
> I just wanted to bring up an issue, possible solution and get feedback 
> on it from folks because it seems to be an on-going problem that shows 
> up not when an application is initially deployed but as on-going 
> operation and running of that application proceeds (ie after running for 
> a period of time).
> 
> The jist of the problem is the following:
> 
> A <> has a need to ensure that no 
> application on the same machine can manipulate a given resource on that 
> same machine, so it uses the lock file pattern (acquire a *local* lock 
> file for that resource, manipulate that resource, release that lock 
> file) to do actions on that resource in a safe manner (note this does 
> not ensure safety outside of that machine, lock files are *not* 
> distributed locks).
> 
> The api that we expose from oslo is typically accessed via the following:
> 
>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None, 
> external=False, lock_path=None, semaphores=None, delay=0.01)
> 
> or via its underlying library (that I extracted from oslo.concurrency 
> and have improved to add more usefulness) @ 
> http://fasteners.readthedocs.org/
> 
> The issue though for <> is that each of 
> these projects now typically has a large amount of lock files that exist 
> or have existed and no easy way to determine when those lock files can 
> be deleted (afaik no? periodic task exists in said projects to clean up 
> lock files, or to delete them when they are no longer in use...) so what 
> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387 
> appear and there is no a simple solution to clean lock files up (since 
> oslo.concurrency is really not the right layer to know when a lock can 
> or can not be deleted, only the application knows that...)
> 
> So then we get a few creative solutions like the following:
> 
> - https://review.openstack.org/#/c/241663/
> - https://review.openstack.org/#/c/239678/
> - (and others?)
> 
> So I wanted to ask the question, how are people involved in < favorite openstack project>> cleaning up these files (are they at all?)
> 
> Another idea that I have been proposing also is to use offset locks.
> 
> This would allow for not creating X lock files, but create a *single* 
> lock file per project and use offsets into it as the way to lock. For 
> example nova could/would create a 1MB (or larger/smaller) *empty* file 
> for locks, that would allow for 1,048,576 locks to be used at the same 
> time, which honestly should be way more than enough, and then there 
> would not need to be any lock cleanup at all... Is there any reason this 
> wasn't initially done back way when this lock file code was created? 
> (https://github.com/harlowja/fasteners/pull/10 adds this functionality 
> to the underlying library if people want to look it over)

I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that.  Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions?  First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
which allows consumers to delete lock files when they're done with them.
 Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.

Second, is this actually a problem?  Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either.  In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files.  At that point you know it's safe to
delete them, and it prevents your lock file directory from growing forever.

I know we've had this discussion in the past, but I don't think anyone
has ever told me that having lock files hang around was a functional
problem for them.  It seems to be largely cosmetic complaints about not
cleaning up the old files (which, as you noted, Oslo can't really solve
because we have no idea when consumers are finished with locks) and
given the amount of trouble we've had with interprocess locking in the
past I've never felt that a cosmetic issue was sufficient reason to
reopen that can of worms.  I'll just note again that every time we've
started messing with this stuff we run into a bunch of sticky problems
and edge cases, so it would take a pretty 

Re: [openstack-dev] [TripleO/heat] openstack debug command

2015-11-30 Thread Steve Baker

On 30/11/15 23:21, Steven Hardy wrote:

On Mon, Nov 30, 2015 at 10:03:29AM +0100, Lennart Regebro wrote:

I'm tasked to implement a command that shows error messages when a
deployment has failed. I have a vague memory of having seen scripts
that do something like this, if that exists, can somebody point me in
teh right direction?

I wrote a super simple script and put it in a blog post a while back:

http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-3-cluster.html

All it does is find the failed SoftwareDeployment resources, then do heat
deployment-show on the resource, so you can see the stderr associated with
the failure.

Having tripleoclient do that by default would be useful.


Any opinions on what that should do, specifically? Traverse failed
resources to find error messages, I assume. Anything else?

Yeah, but I think for this to be useful, we need to go a bit deeper than
just showing the resource error - there are a number of typical failure
modes, and I end up repeating the same steps to debug every time.

1. SoftwareDeployment failed (mentioned above).  Every time, you need to
see the name of the SoftwareDeployment which failed, figure out if it
failed on one or all of the servers, then look at the stderr for clues.

2. A server failed to build (OS::Nova::Server resource is FAILED), here we
need to check both nova and ironic, looking first to see if ironic has the
node(s) in the wrong state for scheduling (e.g nova gave us a no valid
host error), and then if they are OK in ironic, do nova show on the failed
host to see the reason nova gives us for it failing to go ACTIVE.

3. A stack timeout happened.  IIRC when this happens, we currently fail
with an obscure keystone related backtrace due to the token expiring.  We
should instead catch this error and show the heat stack status_reason,
which should say clearly the stack timed out.

If we could just make these three cases really clear and easy to debug, I
think things would be much better (IME the above are a high proportion of
all failures), but I'm sure folks can come up with other ideas to add to
the list.

I'm actually drafting a spec which includes a command which does this. I 
hope to submit it soon, but here is the current state of that command's 
description:


Diagnosing resources in a FAILED state
--

One command will be implemented:
- openstack overcloud failed list

This will print a yaml tree showing the hierarchy of nested stacks until it
gets to the actual failed resource, then it will show information 
regarding the

failure. For most resource types this information will be the status_reason,
but for software-deployment resources the deploy_stdout, deploy_stderr and
deploy_status code will be printed.

In addition to this stand-alone command, this output will also be 
printed when

an ``openstack overcloud deploy`` or ``openstack overcloud update`` command
results in a stack in a FAILED state.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Encouraging first-time contributors through bug tags/reviews

2015-11-30 Thread sean roberts
Being successful at your first patch for most people means that their first
effort is different than an a regular patch.

Identifying abandoned work more quickly is good. It doesn't help the first
timer.

Tagging low hanging fruit for first timers I like. I'm recommending we add
a mentor as part of the idea, so the project mentor is responsible for the
work and the first timer learning.

On Monday, November 30, 2015, Doug Hellmann  wrote:

> Excerpts from sean roberts's message of 2015-11-30 07:57:54 -0800:
> > How about:
> > First timers assign a bug to a mentor and the mentor takes responsibility
> > for the first timer learning from the bug to completion.
>
> That would mean the learning process is different from what we want the
> regular process to be.
>
> If the problem is identifying "In Progress" bugs that are actually not
> being worked on, then let's figure out a way to make that easier.
> sdague's point about the auto-abandon process may help. We could query
> gerrit for "stale" reviews that would have met the old abandon
> requirements and that refer to bugs, for example. Using that
> information, someone could follow-up with the patch owner to see if it
> is actually abandoned, before changing the bug status or encouraging the
> owner to abandon the patch.
>
> >
> > Per project, a few people volunteer themselves as mentors. As easy as
> > responding to [project][mentor] emails.
> >
> > On Monday, November 30, 2015, Sean Dague >
> wrote:
> >
> > > On 11/25/2015 03:22 PM, Shamail wrote:
> > > > Hi,
> > > >
> > > >> On Nov 25, 2015, at 11:05 PM, Doug Hellmann  
> > > > wrote:
> > > >>
> > > >> Excerpts from Shamail Tahir's message of 2015-11-25 09:15:54 -0500:
> > > >>> Hi everyone,
> > > >>>
> > > >>> Andrew Mitry recently shared a medium post[1] by Kent C. Dobbs
> which
> > > >>> discusses how one open-source project is encouraging contributions
> by
> > > new
> > > >>> open-source contributors through a combination of a special tag
> (which
> > > is
> > > >>> associated with work that is needed but can only be completed by
> > > someone
> > > >>> who is a first-time contributor) and helpful comments in the review
> > > phase
> > > >>> to ensure the contribution(s) eventually get merged.
> > > >>>
> > > >>> While reading the article, I immediately thought about our
> > > >>> low-hanging-fruit bug tag which is used for a very similar purpose
> in
> > > "bug
> > > >>> fixing" section of  the "how to contribute" page[2].  The
> > > low-hanging-fruit
> > > >>> tag is used to identify items that are generally suitable for
> > > first-time or
> > > >>> beginner contributors but, in reality, anyone can pick them up.
> > > >>>
> > > >>> I wanted to propose a new tag (or even changing the, existing,
> > > low-hanging
> > > >>> fruit tag) that would identify items that we are reserving for
> > > first-time
> > > >>> OpenStack contributors (e.g. a patch-set for the item submitted by
> > > someone
> > > >>> who is not a first time contributor would be rejected)... The same
> > > article
> > > >>> that Andrew shared mentions using an "up-for-grabs" tag which also
> > > >>> populates the items at up-for-grabs[3] (a site where people
> looking to
> > > >>> start working on open-source projects see entry-level items from
> > > multiple
> > > >>> projects).  If we move forward with an exclusive tag for
> first-timers
> > > then
> > > >>> it would be nice if we could use the up-for-grabs tag so that
> OpenStack
> > > >>> also shows up on the list too.  Please let me know if this change
> > > should be
> > > >>> proposed elsewhere, the tags are maintained in launchpad and the
> wiki I
> > > >>> found related to bug tags[4] didn't indicate a procedure for
> > > submitting a
> > > >>> change proposal.
> > > >>
> > > >> I like the idea of making bugs suitable for first-timers more
> > > >> discoverable. I'm not sure we need to *reserve* any bugs for any
> class
> > > >> of contributor. What benefit do you think that provides?
> > > > I would have to defer to additional feedback here...
> > > >
> > > > My own perspective from when I was doing my first contribution is
> that
> > > it was hard to find active "low-hanging-fruit" items.  Most were
> already
> > > work-in-progress or assigned.
> > >
> > > This was a direct consequence of us dropping the auto-abandoning of old
> > > code reviews in gerrit. When a review is abandoned the bug is flipped
> > > back to New instead of In Progress.
> > >
> > > I found quite often people go and gobble up bugs assigning them to
> > > themselves, but don't make real progress on them. Then new contributors
> > > show up, and don't work on any of those issues because our tools say
> > > someone is already on top of it.
> > >
> > > -Sean
> > >
> > > --
> > > Sean Dague
> > > http://dague.net
> > >
> > >
> 

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Joshua Harlow

Ben Nemec wrote:

On 11/30/2015 12:42 PM, Joshua Harlow wrote:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<>  has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<>  is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<>  cleaning up these files (are they at all?)

Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)


I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that.  Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions?  First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
which allows consumers to delete lock files when they're done with them.
  Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.


Ya, I wonder how many folks are actually doing this, because the exposed 
API of @synchronized doesn't seem to tell u what file to even delete in 
the first place :-/ perhaps we should make that more accessible so that 
people/consumers of that code could know what to delete...




Second, is this actually a problem?  Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either.  In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files.  At that point you know it's safe to
delete them, and it prevents your lock file directory from growing forever.


Except as we move to never shutting an app down (always online and live 
upgrades and all that jazz), it will have to run more than just on boot, 
but point taken.




I know we've had this discussion in the past, but I don't think anyone
has ever told me that having lock files hang around was a functional
problem for them.  It seems to be largely cosmetic complaints about not
cleaning up the old files (which, as you noted, Oslo can't really solve
because we have no idea when consumers are finished with locks) and
given the 

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Sean Dague
On 11/30/2015 03:01 PM, Robert Collins wrote:
> On 1 December 2015 at 08:37, Ben Nemec  wrote:
>> On 11/30/2015 12:42 PM, Joshua Harlow wrote:
>>> Hi all,
>>>
>>> I just wanted to bring up an issue, possible solution and get feedback
>>> on it from folks because it seems to be an on-going problem that shows
>>> up not when an application is initially deployed but as on-going
>>> operation and running of that application proceeds (ie after running for
>>> a period of time).
>>>
>>> The jist of the problem is the following:
>>>
>>> A <> has a need to ensure that no
>>> application on the same machine can manipulate a given resource on that
>>> same machine, so it uses the lock file pattern (acquire a *local* lock
>>> file for that resource, manipulate that resource, release that lock
>>> file) to do actions on that resource in a safe manner (note this does
>>> not ensure safety outside of that machine, lock files are *not*
>>> distributed locks).
>>>
>>> The api that we expose from oslo is typically accessed via the following:
>>>
>>>oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
>>> external=False, lock_path=None, semaphores=None, delay=0.01)
>>>
>>> or via its underlying library (that I extracted from oslo.concurrency
>>> and have improved to add more usefulness) @
>>> http://fasteners.readthedocs.org/
>>>
>>> The issue though for <> is that each of
>>> these projects now typically has a large amount of lock files that exist
>>> or have existed and no easy way to determine when those lock files can
>>> be deleted (afaik no? periodic task exists in said projects to clean up
>>> lock files, or to delete them when they are no longer in use...) so what
>>> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
>>> appear and there is no a simple solution to clean lock files up (since
>>> oslo.concurrency is really not the right layer to know when a lock can
>>> or can not be deleted, only the application knows that...)
>>>
>>> So then we get a few creative solutions like the following:
>>>
>>> - https://review.openstack.org/#/c/241663/
>>> - https://review.openstack.org/#/c/239678/
>>> - (and others?)
>>>
>>> So I wanted to ask the question, how are people involved in <>> favorite openstack project>> cleaning up these files (are they at all?)
>>>
>>> Another idea that I have been proposing also is to use offset locks.
>>>
>>> This would allow for not creating X lock files, but create a *single*
>>> lock file per project and use offsets into it as the way to lock. For
>>> example nova could/would create a 1MB (or larger/smaller) *empty* file
>>> for locks, that would allow for 1,048,576 locks to be used at the same
>>> time, which honestly should be way more than enough, and then there
>>> would not need to be any lock cleanup at all... Is there any reason this
>>> wasn't initially done back way when this lock file code was created?
>>> (https://github.com/harlowja/fasteners/pull/10 adds this functionality
>>> to the underlying library if people want to look it over)
>>
>> I think the main reason was that even with a million locks available,
>> you'd have to find a way to hash the lock names to offsets in the file,
>> and a million isn't a very large collision space for that.  Having two
>> differently named locks that hashed to the same offset would lead to
>> incredibly confusing bugs.
>>
>> We could switch to requiring the projects to provide the offsets instead
>> of hashing a string value, but that's just pushing the collision problem
>> off onto every project that uses us.
>>
>> So that's the problem as I understand it, but where does that leave us
>> for solutions?  First, there's
>> https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
>> which allows consumers to delete lock files when they're done with them.
>>  Of course, in that case the onus is on the caller to make sure the lock
>> couldn't possibly be in use anymore.
>>
>> Second, is this actually a problem?  Modern filesystems have absurdly
>> large limits on the number of files in a directory, so it's highly
>> unlikely we would ever exhaust that, and we're creating all zero byte
>> files so there shouldn't be a significant space impact either.  In the
>> past I believe our recommendation has been to simply create a cleanup
>> job that runs on boot, before any of the OpenStack services start, that
>> deletes all of the lock files.  At that point you know it's safe to
>> delete them, and it prevents your lock file directory from growing forever.
> 
> Not that high - ext3 (still the default for nova ephemeral
> partitions!) has a limit of 64k in one directory.
> 
> That said, I don't disagree - my thinkis is that we should advise
> putting such files on a tmpfs.

So, I think the issue really is that the named external locks were
originally thought to be handling some pretty sensitive critical
sections. Both cinder / nova have less than 20 such 

Re: [openstack-dev] [Fuel] Getting rid of Docker containers on the Fuel master node

2015-11-30 Thread Fox, Kevin M
Is that because of trying to shoehorn docker containers into rpm's though? I've 
never seen anyone else try and use them that way. Maybe they belong in a docker 
repo like the hub or something openstack.org hosted instead?

Thanks,
Kevin

From: Igor Kalnitsky [ikalnit...@mirantis.com]
Sent: Friday, November 27, 2015 1:22 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Fuel] Getting rid of Docker containers on the 
Fuel master node

Hey Vladimir,

Thanks for your effort on doing this job. Unfortunately we have not so
much time left and FF is coming, so I'm afraid it's become unreal to
make it before FF. Especially if it takes 2-3 days to fix system
tests.


Andrew,

I had the same opinion some time ago, but it was changed because
nobody puts effort to fix our Docker experience. Moreover Docker is
still buggy and we have a plenty of issues such as stale mount points
for instance. Besides, I don't like our upgrade procedure -

1. Install fuel-docker-images.rpm
2. Load images from installed tarball to Docker
3. Re-create containers from new images

Where (2) and (3) are manual steps and breaks idea of "yum update"
delivery approach.

Thanks,
Igor

On Wed, Nov 25, 2015 at 9:43 PM, Andrew Woodward  wrote:
> 
> IMO, removing the docker containers is a mistake v.s. fixing them and using
> them properly. They provide an isolation that is necessary (and that we
> mangle) to make services portable and scaleable. We really should sit down
> and document how we really want all of the services to interact before we
> rip the containers out.
>
> I agree, the way we use containers now still is quite wrong, and brings us
> some negative value, but I'm not sold on stripping them out now just because
> they no longer bring the same upgrades value as before.
> 
>
> My opinion aside, we are rushing into this far to late in the feature cycle.
> Prior to moving forward with this, we need a good QA plan, the spec is quite
> light on that and must receive review and approval from QA. This needs to
> include an actual testing plan.
>
> From the implementation side, we are pushing up against the FF deadline. We
> need to document what our time objectives are for this and when we will no
> longer consider this for 8.0.
>
> Lastly, for those that are +1 on the thread here, please review and comment
> on the spec, It's received almost no attention for something with such a
> large impact.
>
> On Tue, Nov 24, 2015 at 4:58 PM Vladimir Kozhukalov
>  wrote:
>>
>> The status is as follows:
>>
>> 1) Fuel-main [1] and fuel-library [2] patches can deploy the master node
>> w/o docker containers
>> 2) I've not built experimental ISO yet (have been testing and debugging
>> manually)
>> 3) There are still some flaws (need better formatting, etc.)
>> 4) Plan for tomorrow is to build experimental ISO and to begin fixing
>> system tests and fix the spec.
>>
>> [1] https://review.openstack.org/#/c/248649
>> [2] https://review.openstack.org/#/c/248650
>>
>> Vladimir Kozhukalov
>>
>> On Mon, Nov 23, 2015 at 7:51 PM, Vladimir Kozhukalov
>>  wrote:
>>>
>>> Colleagues,
>>>
>>> I've started working on the change. Here are two patches (fuel-main [1]
>>> and fuel-library [2]). They are not ready to review (still does not work and
>>> under active development). Changes are not going to be huge. Here is a spec
>>> [3]. Will keep the status up to date in this ML thread.
>>>
>>>
>>> [1] https://review.openstack.org/#/c/248649
>>> [2] https://review.openstack.org/#/c/248650
>>> [3] https://review.openstack.org/#/c/248814
>>>
>>>
>>> Vladimir Kozhukalov
>>>
>>> On Mon, Nov 23, 2015 at 3:35 PM, Aleksandr Maretskiy
>>>  wrote:



 On Mon, Nov 23, 2015 at 2:27 PM, Bogdan Dobrelya
  wrote:
>
> On 23.11.2015 12:47, Aleksandr Maretskiy wrote:
> > Hi all,
> >
> > as you know, Rally runs inside docker on Fuel master node, so docker
> > removal (good improvement) is a problem for Rally users.
> >
> > To solve this, I'm planning to make native Rally installation on Fuel
> > master node that is running on CentOS 7,
> > and then make a step-by-step instruction how to make this
> > installation.
> >
> > So I hope docker removal will not make issues for Rally users.
>
> I believe the most backwards compatible scenario is to keep the docker
> installed while removing the fuel-* docker things back to the host OS.
> So nothing would prevent user from pulling and running whichever docker
> containers he wants to put on the Fuel master node. Makes sense?
>

 Sounds good


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Joshua Harlow

Joshua Harlow wrote:

Ben Nemec wrote:

On 11/30/2015 12:42 PM, Joshua Harlow wrote:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<> has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the
following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<> is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<> cleaning up these files (are they at all?)


From some simple greps using:

$ echo "Removal usage in" $(basename `pwd`); grep -R 
remove_external_lock_file *


Removal usage in cinder


Removal usage in nova
nova/virt/libvirt/imagecache.py: 
lockutils.remove_external_lock_file(lock_file,


Removal usage in glance


Removal usage in neutron


So me thinks people aren't cleaning any of these up :-/



Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)


I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that. Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions? First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151

which allows consumers to delete lock files when they're done with them.
Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.


Ya, I wonder how many folks are actually doing this, because the exposed
API of @synchronized doesn't seem to tell u what file to even delete in
the first place :-/ perhaps we should make that more accessible so that
people/consumers of that code could know what to delete...



Second, is this actually a problem? Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either. In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files. At that point you know it's safe to
delete them, and it prevents your lock file directory from growing
forever.


Except as we move to never shutting an app down (always online and live
upgrades and all that jazz), it will have to run more than just on boot,
but point taken.



I know 

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Joshua Harlow

Clint Byrum wrote:

Excerpts from Joshua Harlow's message of 2015-11-30 10:42:53 -0800:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<>  has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<>  is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<>  cleaning up these files (are they at all?)

Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)


This is really complicated, and basically just makes the directory of
lock files _look_ clean. But it still leaves each offset stale, and has
to be cleaned anyway.


What do u mean here (out of curiosity), each offset stale? The file 
would basically never change size after startup (pick a large enough 
number, 10 million, 1 trillion billion...) and use it appropriately from 
there on out...




Fasteners already has process locks that use fcntl/flock.

These locks provide enough to allow you to infer things about.  the owner
of the lock file. If there's no process still holding the exclusive lock
when you try to lock it, then YOU own it, and thus control the resource.


Well not really, python doesn't expose the ability to introspect who has 
the handle afaik, I  tried to look into that and it looks like fnctl 
(the C api) might have a way to get it, but u can't really introspect 
that, without as u stated, acquiring the lock yourself... I can try to 
recall more of this investigation when I was trying to add a @owner_pid 
property onto fasteners interprocess lock class but from my simple 
memory the exposed API isn't there in python.




A cron job which tries to flock anything older than ${REASONABLE_TIME}
and deletes them seems fine. Whatever process was trying to interact
with the resource is gone at that point.


Yes, or a periodic thread in the application that can do this in a safe 
manner (using its ability to know exactly what its own apps internals 
are doing...)




Now, anything that needs to safely manage a resource beyond without a
live process will need to keep track of its own state and be idempotent
anyway. IMO this isn't something lock files alone solve well. I believe
you're familiar with a library named taskflow that is supposed to help
write code that does this better ;). Even without taskflow, if you are
trying to do something exclusive without a single process that stays
alive, you need to do _something_ to keep track of state and restart
or revert that flow. That is a state management problem, not a locking
problem.



Agreed. ;)


__
OpenStack Development Mailing List (not for usage questions)

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Joshua Harlow

Sean Dague wrote:

On 11/30/2015 03:01 PM, Robert Collins wrote:

On 1 December 2015 at 08:37, Ben Nemec  wrote:

On 11/30/2015 12:42 PM, Joshua Harlow wrote:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<>  has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<>  is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<>  cleaning up these files (are they at all?)

Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)

I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that.  Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions?  First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
which allows consumers to delete lock files when they're done with them.
  Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.

Second, is this actually a problem?  Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either.  In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files.  At that point you know it's safe to
delete them, and it prevents your lock file directory from growing forever.

Not that high - ext3 (still the default for nova ephemeral
partitions!) has a limit of 64k in one directory.

That said, I don't disagree - my thinkis is that we should advise
putting such files on a tmpfs.


So, I think the issue really is that the named external locks were
originally thought to be handling some pretty sensitive critical
sections. Both cinder / nova have less than 20 such named locks.

Cinder uses a parametrized version for all volume operations -
https://github.com/openstack/cinder/blob/7fb767f2d652f070a20fd70d92585d61e56f3a50/cinder/volume/manager.py#L143


Nova also does something similar in image cache

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Ben Nemec
On 11/30/2015 01:57 PM, Joshua Harlow wrote:
> Ben Nemec wrote:
>> On 11/30/2015 12:42 PM, Joshua Harlow wrote:
>>> Hi all,
>>>
>>> I just wanted to bring up an issue, possible solution and get feedback
>>> on it from folks because it seems to be an on-going problem that shows
>>> up not when an application is initially deployed but as on-going
>>> operation and running of that application proceeds (ie after running for
>>> a period of time).
>>>
>>> The jist of the problem is the following:
>>>
>>> A<>  has a need to ensure that no
>>> application on the same machine can manipulate a given resource on that
>>> same machine, so it uses the lock file pattern (acquire a *local* lock
>>> file for that resource, manipulate that resource, release that lock
>>> file) to do actions on that resource in a safe manner (note this does
>>> not ensure safety outside of that machine, lock files are *not*
>>> distributed locks).
>>>
>>> The api that we expose from oslo is typically accessed via the following:
>>>
>>> oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
>>> external=False, lock_path=None, semaphores=None, delay=0.01)
>>>
>>> or via its underlying library (that I extracted from oslo.concurrency
>>> and have improved to add more usefulness) @
>>> http://fasteners.readthedocs.org/
>>>
>>> The issue though for<>  is that each of
>>> these projects now typically has a large amount of lock files that exist
>>> or have existed and no easy way to determine when those lock files can
>>> be deleted (afaik no? periodic task exists in said projects to clean up
>>> lock files, or to delete them when they are no longer in use...) so what
>>> happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
>>> appear and there is no a simple solution to clean lock files up (since
>>> oslo.concurrency is really not the right layer to know when a lock can
>>> or can not be deleted, only the application knows that...)
>>>
>>> So then we get a few creative solutions like the following:
>>>
>>> - https://review.openstack.org/#/c/241663/
>>> - https://review.openstack.org/#/c/239678/
>>> - (and others?)
>>>
>>> So I wanted to ask the question, how are people involved in<>> favorite openstack project>>  cleaning up these files (are they at all?)
>>>
>>> Another idea that I have been proposing also is to use offset locks.
>>>
>>> This would allow for not creating X lock files, but create a *single*
>>> lock file per project and use offsets into it as the way to lock. For
>>> example nova could/would create a 1MB (or larger/smaller) *empty* file
>>> for locks, that would allow for 1,048,576 locks to be used at the same
>>> time, which honestly should be way more than enough, and then there
>>> would not need to be any lock cleanup at all... Is there any reason this
>>> wasn't initially done back way when this lock file code was created?
>>> (https://github.com/harlowja/fasteners/pull/10 adds this functionality
>>> to the underlying library if people want to look it over)
>>
>> I think the main reason was that even with a million locks available,
>> you'd have to find a way to hash the lock names to offsets in the file,
>> and a million isn't a very large collision space for that.  Having two
>> differently named locks that hashed to the same offset would lead to
>> incredibly confusing bugs.
>>
>> We could switch to requiring the projects to provide the offsets instead
>> of hashing a string value, but that's just pushing the collision problem
>> off onto every project that uses us.
>>
>> So that's the problem as I understand it, but where does that leave us
>> for solutions?  First, there's
>> https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
>> which allows consumers to delete lock files when they're done with them.
>>   Of course, in that case the onus is on the caller to make sure the lock
>> couldn't possibly be in use anymore.
> 
> Ya, I wonder how many folks are actually doing this, because the exposed 
> API of @synchronized doesn't seem to tell u what file to even delete in 
> the first place :-/ perhaps we should make that more accessible so that 
> people/consumers of that code could know what to delete...

I'm not opposed to allowing users to clean up lock files, although I
think the docstrings for the methods should be very clear that it isn't
strictly necessary and it must be done carefully to avoid deleting
in-use files (the existing docstring is actually insufficient IMHO, but
I'm pretty sure I reviewed it when it went in so I have no one else to
blame ;-).

> 
>>
>> Second, is this actually a problem?  Modern filesystems have absurdly
>> large limits on the number of files in a directory, so it's highly
>> unlikely we would ever exhaust that, and we're creating all zero byte
>> files so there shouldn't be a significant space impact either.  In the
>> past I believe our recommendation has been to simply create a cleanup
>> job that runs on boot, 

Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Joshua Harlow

Sean Dague wrote:

On 11/30/2015 03:01 PM, Robert Collins wrote:

On 1 December 2015 at 08:37, Ben Nemec  wrote:

On 11/30/2015 12:42 PM, Joshua Harlow wrote:

Hi all,

I just wanted to bring up an issue, possible solution and get feedback
on it from folks because it seems to be an on-going problem that shows
up not when an application is initially deployed but as on-going
operation and running of that application proceeds (ie after running for
a period of time).

The jist of the problem is the following:

A<>  has a need to ensure that no
application on the same machine can manipulate a given resource on that
same machine, so it uses the lock file pattern (acquire a *local* lock
file for that resource, manipulate that resource, release that lock
file) to do actions on that resource in a safe manner (note this does
not ensure safety outside of that machine, lock files are *not*
distributed locks).

The api that we expose from oslo is typically accessed via the following:

oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
external=False, lock_path=None, semaphores=None, delay=0.01)

or via its underlying library (that I extracted from oslo.concurrency
and have improved to add more usefulness) @
http://fasteners.readthedocs.org/

The issue though for<>  is that each of
these projects now typically has a large amount of lock files that exist
or have existed and no easy way to determine when those lock files can
be deleted (afaik no? periodic task exists in said projects to clean up
lock files, or to delete them when they are no longer in use...) so what
happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
appear and there is no a simple solution to clean lock files up (since
oslo.concurrency is really not the right layer to know when a lock can
or can not be deleted, only the application knows that...)

So then we get a few creative solutions like the following:

- https://review.openstack.org/#/c/241663/
- https://review.openstack.org/#/c/239678/
- (and others?)

So I wanted to ask the question, how are people involved in<>  cleaning up these files (are they at all?)

Another idea that I have been proposing also is to use offset locks.

This would allow for not creating X lock files, but create a *single*
lock file per project and use offsets into it as the way to lock. For
example nova could/would create a 1MB (or larger/smaller) *empty* file
for locks, that would allow for 1,048,576 locks to be used at the same
time, which honestly should be way more than enough, and then there
would not need to be any lock cleanup at all... Is there any reason this
wasn't initially done back way when this lock file code was created?
(https://github.com/harlowja/fasteners/pull/10 adds this functionality
to the underlying library if people want to look it over)

I think the main reason was that even with a million locks available,
you'd have to find a way to hash the lock names to offsets in the file,
and a million isn't a very large collision space for that.  Having two
differently named locks that hashed to the same offset would lead to
incredibly confusing bugs.

We could switch to requiring the projects to provide the offsets instead
of hashing a string value, but that's just pushing the collision problem
off onto every project that uses us.

So that's the problem as I understand it, but where does that leave us
for solutions?  First, there's
https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
which allows consumers to delete lock files when they're done with them.
  Of course, in that case the onus is on the caller to make sure the lock
couldn't possibly be in use anymore.

Second, is this actually a problem?  Modern filesystems have absurdly
large limits on the number of files in a directory, so it's highly
unlikely we would ever exhaust that, and we're creating all zero byte
files so there shouldn't be a significant space impact either.  In the
past I believe our recommendation has been to simply create a cleanup
job that runs on boot, before any of the OpenStack services start, that
deletes all of the lock files.  At that point you know it's safe to
delete them, and it prevents your lock file directory from growing forever.

Not that high - ext3 (still the default for nova ephemeral
partitions!) has a limit of 64k in one directory.

That said, I don't disagree - my thinkis is that we should advise
putting such files on a tmpfs.


So, I think the issue really is that the named external locks were
originally thought to be handling some pretty sensitive critical
sections. Both cinder / nova have less than 20 such named locks.

Cinder uses a parametrized version for all volume operations -
https://github.com/openstack/cinder/blob/7fb767f2d652f070a20fd70d92585d61e56f3a50/cinder/volume/manager.py#L143


Nova also does something similar in image cache

Re: [openstack-dev] [ironic] Third Party CI Deadlines for Mitaka and N

2015-11-30 Thread Anita Kuno
On 11/30/2015 08:00 PM, Mike Perez wrote:
> On October 28th 2015 at the Ironic Third Party CI summit session [1], there 
> was
> consensus by the Ironic core and participating vendors that the set of
> deadlines will be:
> 
> * Mitaka-2ː Driver teams will have registered their intent to run CI by 
> creating
> system accounts and identifying a point of contact for their CI team in the
> Third party CI wiki [2].
> * Mitaka Feature Freezeː All driver systems show the ability to receive events
> and post comments in the sandbox.
> * N release feature freezeː Per patch testing and posting comments.
> 
> There are requirements set for OpenStack Third Party CI's [3]. In addition
> Ironic third party CI's must:
> 
> 1) Test all drivers your company has integrated in Ironic.
> 
> For example, if your company has two drivers in Ironic, you would need to have
> a CI that tests against the two and reports the results for each, for every
> Ironic upstream patch. The tests come from a Devstack Gate job template [4], 
> in
> which you just need to switch the "deploy_driver" to your driver.
> 
> To get started, read OpenStack's third party testing documentation [5]. There
> are efforts by OpenStack Infra to allow others to run third party CI similar 
> to
> the OpenStack upstream CI using Puppet [6] and instruction are available [7].
> Don't forget to register your CI in the wiki [2], there is no need to announce
> about it on any mailing list.
> 
> OpenStack Infra also provides third party CI help via meetings [8], and the
> Ironic team has designated people to answer questions with setting up a third
> party CI in the #openstack-ironic room [9].
> 
> If a solution does not have a CI watching for events and posting comments to
> the sandbox [10] by the Mitaka feature freeze, it'll be assumed the driver is
> not active, and can be removed from the Ironic repository as of the Mitaka
> release.

Thanks Mike, great post.

One point of clarification, the sandbox repo for third-party ci systems
is called ci-sandbox:
https://review.openstack.org/#/q/project:+openstack-dev/ci-sandbox,n,z
also found here: http://git.openstack.org/cgit/openstack-dev/ci-sandbox/

The sandbox linked in the original post is for developers to experiment
with Gerrit not for ci systems.

Thank you,
Anita.

> 
> If a solution is not being tested in a CI system and reporting to OpenStack
> gerrit Ironic patches by the deadline of the N release feature freeze, an
> Ironic driver could be removed from the Ironic repository. Without a CI 
> system,
> Ironic core is unable to verify your driver works in the N release of Ironic.
> 
> If there is something not clear about this email, please email me *directly*
> with your question. You can also reach me as thingee on Freenode IRC in the
> #openstack-ironic channel. Again I want you all to be successful in this, and
> take advantage of this testing you will have with your product. Please
> communicate with me and reach out to the team for help.
> 
> [1] - https://etherpad.openstack.org/p/summit-mitaka-ironic-third-party-ci
> [2] - https://wiki.openstack.org/wiki/ThirdPartySystems
> [3] - 
> http://docs.openstack.org/infra/system-config/third_party.html#requirements
> [4] - 
> https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L961
> [5] - http://docs.openstack.org/infra/system-config/third_party.html
> [6] - https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/
> [7] - 
> https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md
> [8] - https://wiki.openstack.org/wiki/Meetings/ThirdParty
> [9] - https://wiki.openstack.org/wiki/Ironic/Testing#Questions
> [10] - https://review.openstack.org/#/q/project:+openstack-dev/sandbox,n,z
> 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [cross-project] Cross-project Liaisons

2015-11-30 Thread Mike Perez
Hello all,

Currently for cross-project specs, the author of the spec spends the time to
explain why a certain feature makes sense to be across multiple projects. This
also includes giving technical solutions for it working with a variety of
services and making sure everyone is happy.

Today we have the following problems:

* Authors of specs can't progress forward with specs because of lack of
  attention. Eventually getting frustrated and giving up.
* Some projects could miss a cross-project spec being approved by the TC.

It has been expressed to me at the previous Cross-Project Communication Tokyo
summit session that PTLs don't have time for cross-project issues. I agree, as
being a previous PTL your time is thin. However, I do think someone from each
project needs to be aware and involved with cross-project initiatives.

I would like to propose cross-project liaisons which would have the following
duties:

* Watching the cross-project spec repo [1].
  - Comment on specs that involve your project. +1 to carry forward for TC
approval.
-- If you're not able to provide technical guidance on certain specs for
   your project, it's up to you to get the right people involved.
-- Assuming you get someone else involved, it's up to you to make sure they
   keep up with communication.
  - Communicate back to your project's meeting on certain cross-project specs
when necessary. This is also good for the previous bullet point of sourcing
who would have technical knowledge for certain specs.
* Attend the cross-project meeting when it's called for [2].


[1] - 
https://review.openstack.org/#/q/project:+openstack/openstack-specs+status:+open,n,z
[2] - https://wiki.openstack.org/wiki/Meetings/CrossProjectMeeting

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [midonet] midonet weekly irc meeting

2015-11-30 Thread Ryu Ishimoto
Hi All!

We would like to start holding a weekly IRC meeting for midonet, and
we want to propose the meeting time of 9:00 UTC on Tuesdays.

Please let me know if anyone prefers different time.

Best,
Ryu

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [telemetry][aodh] The purpose of notification about alarm updating

2015-11-30 Thread liusheng

Hi folks,

Currently, a notification message will be emitted when updating an alarm 
(state transition, attribute updating, creation),  this functionality 
was added by change[1], but the change didn't describe any purpose. So I 
wonder whether there is any usage of this type of notification, we can 
get the whole details about alarm change by alarm-history API. the 
notification is implicitly ignored by default, because the 
"notification_driver"config option won't be configured by default.  if 
we enable this option in aodh.conf and enable the "store_events" in 
ceilometer.conf, this type of notifications will be stored as events. so 
maybe some users want to aggregate this with events ? what's your opinion ?


I have made a change try to deprecate this notification, see [2].

[1] https://review.openstack.org/#/c/48949/
[2] https://review.openstack.org/#/c/246727/

BR
Liu sheng
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Cross-Project meeting, Tue Dec 1st, 21:00 UTC & *New Location*

2015-11-30 Thread Mike Perez
Dear PTLs, cross-project liaisons and anyone else interested,

We'll have a cross-project meeting December 1st at 21:00 UTC in the NEW
#openstack-meeting-cp IRC channel, with the following agenda: 

* Review past action items
* Team announcements (horizontal, vertical, diagonal)
* backwards compat of libraries and clients [1]
* Cross-project Liaisons [2]
* Open discussion

If you're from a horizontal team (Release management, QA, Infra, Docs,
Security, I18n...) or a vertical team (Nova, Swift, Keystone...) and
have something to communicate to the other teams, feel free to abuse the
relevant sections of that meeting and make sure it gets #info-ed by the
meetbot in the meeting summary.

See you there!

For more details on this meeting, please see:
https://wiki.openstack.org/wiki/Meetings/CrossProjectMeeting

[1] - https://review.openstack.org/226157
[2] - 
http://lists.openstack.org/pipermail/openstack-dev/2015-December/080869.html

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO/heat] openstack debug command

2015-11-30 Thread Steve Baker

On 01/12/15 10:28, Steven Hardy wrote:

On Tue, Dec 01, 2015 at 08:47:20AM +1300, Steve Baker wrote:

On 30/11/15 23:21, Steven Hardy wrote:

On Mon, Nov 30, 2015 at 10:03:29AM +0100, Lennart Regebro wrote:

I'm tasked to implement a command that shows error messages when a
deployment has failed. I have a vague memory of having seen scripts
that do something like this, if that exists, can somebody point me in
teh right direction?

I wrote a super simple script and put it in a blog post a while back:

http://hardysteven.blogspot.co.uk/2015/05/tripleo-heat-templates-part-3-cluster.html

All it does is find the failed SoftwareDeployment resources, then do heat
deployment-show on the resource, so you can see the stderr associated with
the failure.

Having tripleoclient do that by default would be useful.


Any opinions on what that should do, specifically? Traverse failed
resources to find error messages, I assume. Anything else?

Yeah, but I think for this to be useful, we need to go a bit deeper than
just showing the resource error - there are a number of typical failure
modes, and I end up repeating the same steps to debug every time.

1. SoftwareDeployment failed (mentioned above).  Every time, you need to
see the name of the SoftwareDeployment which failed, figure out if it
failed on one or all of the servers, then look at the stderr for clues.

2. A server failed to build (OS::Nova::Server resource is FAILED), here we
need to check both nova and ironic, looking first to see if ironic has the
node(s) in the wrong state for scheduling (e.g nova gave us a no valid
host error), and then if they are OK in ironic, do nova show on the failed
host to see the reason nova gives us for it failing to go ACTIVE.

3. A stack timeout happened.  IIRC when this happens, we currently fail
with an obscure keystone related backtrace due to the token expiring.  We
should instead catch this error and show the heat stack status_reason,
which should say clearly the stack timed out.

If we could just make these three cases really clear and easy to debug, I
think things would be much better (IME the above are a high proportion of
all failures), but I'm sure folks can come up with other ideas to add to
the list.


I'm actually drafting a spec which includes a command which does this. I
hope to submit it soon, but here is the current state of that command's
description:

Diagnosing resources in a FAILED state
--

One command will be implemented:
- openstack overcloud failed list

This will print a yaml tree showing the hierarchy of nested stacks until it
gets to the actual failed resource, then it will show information regarding
the
failure. For most resource types this information will be the status_reason,
but for software-deployment resources the deploy_stdout, deploy_stderr and
deploy_status code will be printed.

In addition to this stand-alone command, this output will also be printed
when
an ``openstack overcloud deploy`` or ``openstack overcloud update`` command
results in a stack in a FAILED state.

This sounds great!

The spec is here.

Another piece of low-hanging-fruit in the meantime is we should actually
print the stack_status_reason on failure:

https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L280

The DeploymentError raised could include the stack_status_reason vs the
unqualified "Heat Stack create failed".

I guess your event listing partially overlaps with this, as you can now
derive the stack_status_reason from the last event, but it's still be good
to loudly output it so folks can see more quickly when things such as
timeouts happen that are clearly displayed in the top-level stack status.


Yes, this would be a trivially implemented quick win.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread hao wang
Thank you Walter for those information. Yes, I knew Smaug, actually I
was one of speakers in this session :).
This project was started by my colleague, Eran Gampel.
AFAIK, resources in Cinder(volumes, etc.) is considered as one of
protection objects in it.

So we can see there are more and more attention to DR,  as discussed
here, it's a complex issue, so I feel cinder will
provide basic ability to support DR, and Smaug or DRagon, will use
those APIs(and Nova's APIs, Heat's APIs, etc.) to
implement the DR goal for OpenStack resources.

2015-12-01 0:50 GMT+08:00 Walter A. Boring IV :
> As a side note to the DR discussion here, there was a session in Tokyo that
> talked about a new
> DR project called Smaug.   You can see their mission statement here:
> https://launchpad.net/smaug
>
> https://github.com/openstack/smaug
>
> There is another service in the making called DRagon:
> https://www.youtube.com/watch?v=upCzuFnswtw
> http://www.slideshare.net/AlonMarx/dragon-and-cinder-v-brownbag-54639869
>
> Yes that's 2 DR like service starting in OpenStack that are related to
> dragons.
>
> Walt
>
>
>> Sean and Michal,
>>
>> In fact, there is a reason that I ask this question. Recently I have a
>> confusion about if cinder should provide the ability of Disaster
>> Recovery to storage resources, like volume. I mean we have volume
>> replication v1, but for DR, specially DR between two independent
>> OpenStack sites(production and DR site), I feel we still need more
>> features to support it, for example consistency group for replication,
>> etc. I'm not sure if those features belong in Cinder or some new
>> project for DR.
>>
>> BR
>> WangHao
>>
>> 2015-11-30 3:02 GMT+08:00 Sean McGinnis :
>>>
>>> On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:

 On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
>
> Hi guys,
>
> I notice nova have a clarification of project scope:
> http://docs.openstack.org/developer/nova/project_scope.html
>
> I want to find cinder's, but failed,  do you know where to find it?
>
> It's important to let developers know what feature should be
> introduced into cinder and what shouldn't.
>
> BR
> Wang Hao

 I believe Nova team needed to formalize the scop to have an explanation
 for all the "this doesn't belong in Nova" comments on feature requests.
 Does Cinder suffer from similar problems? From my perspective it's not
 critically needed.
>>>
>>> I agree. I haven't seen a need for something like that with Cinder. Wang
>>> Hao, is there a reason you feel you need that?
>>>
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> .
>>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Steven Hardy
On Mon, Nov 30, 2015 at 12:51:53PM -0500, Ruby Loo wrote:
>On 30 November 2015 at 10:19, Derek Higgins  wrote:
> 
>  Hi All,
> 
>      A few months tripleo switch from its devtest based CI to one that
>  was based on instack. Before doing this we anticipated disruption in the
>  ci jobs and removed them from non tripleo projects.
> 
>      We'd like to investigate adding it back to heat and ironic as
>  these are the two projects where we find our ci provides the most value.
>  But we can only do this if the results from the job are treated as
>  voting.
> 
>What does this mean? That the tripleo job could vote and do a -1 and block
>ironic's gate?

I believe it means they would be non voting, but cores should be careful
not to ignore them, e.g if a patch isn't passing tripleo CI it should be
investigated before merging said patch.

>      In the past most of the non tripleo projects tended to ignore the
>  results from the tripleo job as it wasn't unusual for the job to broken
>  for days at a time. The thing is, ignoring the results of the job is the
>  reason (the majority of the time) it was broken in the first place.
>      To decrease the number of breakages we are now no longer running
>  master code for everything (for the non tripleo projects we bump the
>  versions we use periodically if they are working). I believe with this
>  model the CI jobs we run have become a lot more reliable, there are
>  still breakages but far less frequently.
> 
>  What I proposing is we add at least one of our tripleo jobs back to both
>  heat and ironic (and other projects associated with them e.g. clients,
>  ironicinspector etc..), tripleo will switch to running latest master of
>  those repositories and the cores approving on those projects should wait
>  for a passing CI jobs before hitting approve. So how do people feel
>  about doing this? can we give it a go? A couple of people have already
>  expressed an interest in doing this but I'd like to make sure were all
>  in agreement before switching it on.
> 
>This seems to indicate that the tripleo jobs are non-voting, or at least
>won't block the gate -- so I'm fine with adding tripleo jobs to ironic.
>But if you want cores to wait/make sure they pass, then shouldn't they be
>voting? (Guess I'm a bit confused.)

The subtext here is that automated testing of OpenStack deployments is
hard, and TripleO CI sometimes experiences breakage for various reasons
including regressions in any one of the OpenStack projects it uses.

For example, TripleO CI has been broken for the last day or two due to a
nodepool regression - in this scenario it's probably best for Ironic and
Heat cores to maintain the ability to land patches, even if we may decide
it's unwise to land larger and/or more risky changes until they can be
validated against TripleO CI.

Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Devananda van der Veen
On Mon, Nov 30, 2015 at 3:07 PM, Zane Bitter  wrote:

> On 30/11/15 12:51, Ruby Loo wrote:
>
>>
>>
>> On 30 November 2015 at 10:19, Derek Higgins > > wrote:
>>
>> Hi All,
>>
>>  A few months tripleo switch from its devtest based CI to one
>> that was based on instack. Before doing this we anticipated
>> disruption in the ci jobs and removed them from non tripleo projects.
>>
>>  We'd like to investigate adding it back to heat and ironic as
>> these are the two projects where we find our ci provides the most
>> value. But we can only do this if the results from the job are
>> treated as voting.
>>
>>
>> What does this mean? That the tripleo job could vote and do a -1 and
>> block ironic's gate?
>>
>>
>>  In the past most of the non tripleo projects tended to ignore
>> the results from the tripleo job as it wasn't unusual for the job to
>> broken for days at a time. The thing is, ignoring the results of the
>> job is the reason (the majority of the time) it was broken in the
>> first place.
>>  To decrease the number of breakages we are now no longer
>> running master code for everything (for the non tripleo projects we
>> bump the versions we use periodically if they are working). I
>> believe with this model the CI jobs we run have become a lot more
>> reliable, there are still breakages but far less frequently.
>>
>> What I proposing is we add at least one of our tripleo jobs back to
>> both heat and ironic (and other projects associated with them e.g.
>> clients, ironicinspector etc..), tripleo will switch to running
>> latest master of those repositories and the cores approving on those
>> projects should wait for a passing CI jobs before hitting approve.
>> So how do people feel about doing this? can we give it a go? A
>> couple of people have already expressed an interest in doing this
>> but I'd like to make sure were all in agreement before switching it
>> on.
>>
>> This seems to indicate that the tripleo jobs are non-voting, or at least
>> won't block the gate -- so I'm fine with adding tripleo jobs to ironic.
>> But if you want cores to wait/make sure they pass, then shouldn't they
>> be voting? (Guess I'm a bit confused.)
>>
>
> +1
>
> I don't think it hurts to turn it on, but tbh I'm uncomfortable with the
> mental overhead of a non-voting job that I have to manually treat as a
> voting job. If it's stable enough to make it a voting job, I'd prefer we
> just make it voting. And if it's not then I'd like to see it be made stable
> enough to be a voting job and then make it voting.


This is roughly where I sit as well -- if it's non-voting, experience tells
me that it will largely be ignored, and as such, isn't a good use of
resources.

I haven't looked at tripleo or tripleoci in a while, so I wont assume that
my recollection of the CI jobs bears any resemblance to what exists today.
Could you explain what areas of ironic (or its subprojects) will be covered
by these tests?  If they are already covered by existing tests, then I
don't see the benefit of adding another job; conversely, if this is testing
areas we don't cover today, then there's probably value in running
tripleoci in a voting fashion for now and then moving that coverage into
ironic's project testing.

-Deva
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [QA][Tempest] Use tempest-config for tempest-cli-improvements

2015-11-30 Thread David_Paterson
Dell - Internal Use - Confidential
 I don’t really understand the friction, it’s incredibly useful, especially for 
new users.

Use it or don’t use it it’s up to you,  nowhere is it implied that the user is 
forced to configure tempest with the tool.

Without a tool like this it is complex for new users to get up and going with 
tempest.  It is also more difficult for other parties to integrate with 
Tempest, Rally had to write their own configuration tooling for instance.

Thanks,
dp



From: Andrea Frittoli [mailto:andrea.fritt...@gmail.com]
Sent: Monday, November 30, 2015 8:34 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [QA][Tempest] Use tempest-config for 
tempest-cli-improvements

Looking at the tool, it seems to me that is servers a combination of functions:
- provision test resources
- support for distribution specific / cloud specific overrides to default
- support for configuration override via CLI
- discovery of configuration

Test resource provisioning is something that I agree is useful to have.
We plan in Mitaka to separate out of tempest.conf the configuration of test 
resources, and have a CLI tool to provision them [0]. We could re-use code from 
this tool to achieve that.

Support for distribution specific / cloud specific overrides is also something 
that is useful. In clouds where I control the deployment process I inject extra 
configs in tempest.conf based on the deployment options. In clouds where I 
don't, I maintain a partial tempest.conf with the list of options which I 
expects I have to modify to match the target cloud.

That's pretty easy to achieve though - simply append the extra configs to the 
bottom of tempest.conf and it's done. Duplicate configuration options are not 
an issue, the last one wins. Still we could support specifying a number of 
configuration files to the non-yet implemented "tempest run" command.

Support for configuration override via CLI is something that I can see it can 
be useful during development or troubleshooting, we could support that as 
options of the non-yet implemented "tempest run" command.

The last point is discovery - I believe we should use that only as we use it 
today in the gate - i.e. fail fast if the generated configuration does not 
match what can be discovered from the cloud.

So I would not get the script as is into tempest, but I think many of the 
functions implemented by it can fit into tempest - and some are already there.

andrea

[0] https://review.openstack.org/#/c/173334/

On Mon, Nov 30, 2015 at 7:39 AM Masayuki Igawa 
> wrote:
Hi,

I agree with Ken'ichi's opinion, basically. Tempest users should know "what do 
we test for?" and we shouldn't discover values that we test for automatically.
If users seems that "My current cloud is good. This is what I expect.", 
discovering function could work. But I suppose many of users would use 
tempest-config-generator for a new their cloud. So I feel the tool users could 
be misunderstanding easily.

But I also think that tempest users don't need to know all of the 
configurations.
So, how about something like introducing "a configuration wizard" for tempest 
configuration?
This is a just idea, though..


Anyway, if you have the motivation to introduce tempest-config, how about 
writing a spec for the feature for a concrete discussion?
(I think we should have agreements about the target issues, users, solutions, 
etc.)

Best Regards,
-- Masayuki Igawa

2015年11月29日(日) 22:07 Yair Fried >:
Hi,
I agree with Jordan.
We don't have to use the tool as part of the gate. It's target audience is 
people and not CI systems. More specifically - new users.
However, we could add a gate (or a few) for the tool that makes sure a proper 
conf file is generated. It doesn't have to run the tests, just compare the 
output of the script to the conf generated by devstack.

Re Rally - I believe the best place for tempest configuration script is within 
tempest. That said, if the Tempest community doesn't want this tool, we'll have 
to settle for the Rally solution.

Regards
Yair

On Fri, Nov 27, 2015 at 11:31 AM, Jordan Pittier 
> wrote:
Hi,
I think this script is valuable to some users: Rally and Red Hat expressed 
their needs, they seem clear.

This tool is far from bullet proof and if used blindly or in case of bugs, 
Tempest could be misconfigured. So, we could have this tool inside the Tempest 
repository (in the tools/) but not use it at all for the Gate.

I am not sure I fully understand the resistance for this, if we don"t use this 
config generator for the gate, what's the risk ?

Jordan

On Fri, Nov 27, 2015 at 8:05 AM, Ken'ichi Ohmichi 
> wrote:
2015-11-27 15:40 GMT+09:00 Daniel Mellado 

[openstack-dev] [Neutron] Evolving the stadium concept

2015-11-30 Thread Armando M.
Hi folks,

The stadium concept was introduced more or less formally since April of
this year. At the time it was introduced (see [1]), the list of
deliverables included neutron, client, specs and *-aas services. As you may
be well aware, 6+ months is a long time in the OpenStack world, and lots of
things happened since then. The list has grown to [2].

When I think about what 'deliverables' are, I am inclined to think that all
of the projects that are part of the list will have to behave and follow
the same rules, provided that there is flexibility given by the tags. However,
reality has proven us that rules are somewhat difficult to follow and
enforce, and some boundaries may be too strict for some initiatives to
comply with. This is especially true if we go from a handful of projects
that we had when this had started to the nearly the two dozens we have now.

As a result, there is quite an effort imposed on the PTL, the various
liaisons (release, infra, docs, testing, etc) and the core team to help
manage the existing relationships and to ensure that the picture stays
coherent over time. Sometimes the decision of being part of this list is
even presented before one can see any code, and that defeats the whole
point of the deliverable association. I have experienced first hand that
this has become a burden, and I fear that the stadium might be an extra
layer of governance/complexity that could even interfere with the existing
responsibilities of the TC and of OpenStack infra.

So my question is: would revisiting/clarifying the concept be due after
some time we have seen it in action? I would like to think so. To be fair,
I am not sure what the right answer is, but I know for a fact that some
iterations are in order, and I like to make a proposal:

I would like to suggest that we evolve the structure of the Neutron
governance, so that most of the deliverables that are now part of the
Neutron stadium become standalone projects that are entirely self-governed
(they have their own core/release teams, etc). In order to denote the
initiatives that are related to Neutron I would like to present two new
tags that projects can choose to label themselves with:


   - 'is-neutron-subsystem': this means that the project provides
   networking services by implementing an integral part (or parts) of an
   end-to-end neutron system. Examples are: a service plugin, an ML2 mech
   driver, a monolithic plugin, an agent etc. It's something an admin has to
   use in order to deploy Neutron in a certain configuration.
   - 'use-neutron-system': this means that the project provides networking
   services by using a pre-deployed end-to-end neutron system as is. No
   modifications whatsoever.

As a result, there is no oversight by the Neutron core team, the PTL or
liasons, but that should not stop people from being involved if they choose
to. We would not lose the important piece of information which is the
association to Neutron, and at the same time that would relieve some of us
from the onus of dealing with initiatives for which we lack enough context
and ability of providing effective guidance.

In the process, the core team should stay focused on breaking the coupling
that still affects Neutron so that projects that depend on it can do so
more reliably, and yet innovate more independently. If that means
revisiting the Neutron's mission statement, we can discuss that too.

I am sure this hardly covers all the questions you may have at this point,
but I would like to take the opportunity to start the conversation to see
where people stand. Whichever the outcome, I think that we should strive
for decentralizing responsibilities as much as we can in order to be
scalable as a project and I think that the current arrangement prevents us
from doing that.

Thanks for reading.

Armando

[1]
https://github.com/openstack/governance/blob/april-2015-elections/reference/projects.yaml#L141
[2]
https://github.com/openstack/governance/blob/master/reference/projects.yaml#L2000
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [nova][glance][cinder][neutron]How to make use of x-openstack-request-id

2015-11-30 Thread Tan, Lin
Hi guys
I recently play around with 'x-openstack-request-id' header but have a 
dump question about how it works. At beginning, I thought an action across 
different services should use a same request-id but it looks like this is not 
the true. 

First I read the spec: 
https://blueprints.launchpad.net/nova/+spec/cross-service-request-id which said 
"This ID and the request ID of the other service will be logged at service 
boundaries". and I see cinder/neutron/glance will attach its context's 
request-id as the value of "x-openstack-request-id" header to its response 
while nova use X-Compute-Request-Id. This is easy to understand. So It looks 
like each service should generate its own request-id and attach to its 
response, that's all.

But then I see glance read 'X-Openstack-Request-ID' to generate the request-id 
while cinder/neutron/nova read 'openstack.request_id' when using with keystone. 
It is try to reuse the request-id from keystone.

This totally confused me. It would be great if you can correct me or point me 
some reference. Thanks a lot

Best Regards,

Tan


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo][all] The lock files saga (and where we can go from here)

2015-11-30 Thread Clint Byrum
Excerpts from Ben Nemec's message of 2015-11-30 13:22:23 -0800:
> On 11/30/2015 02:15 PM, Sean Dague wrote:
> > On 11/30/2015 03:01 PM, Robert Collins wrote:
> >> On 1 December 2015 at 08:37, Ben Nemec  wrote:
> >>> On 11/30/2015 12:42 PM, Joshua Harlow wrote:
>  Hi all,
> 
>  I just wanted to bring up an issue, possible solution and get feedback
>  on it from folks because it seems to be an on-going problem that shows
>  up not when an application is initially deployed but as on-going
>  operation and running of that application proceeds (ie after running for
>  a period of time).
> 
>  The jist of the problem is the following:
> 
>  A <> has a need to ensure that no
>  application on the same machine can manipulate a given resource on that
>  same machine, so it uses the lock file pattern (acquire a *local* lock
>  file for that resource, manipulate that resource, release that lock
>  file) to do actions on that resource in a safe manner (note this does
>  not ensure safety outside of that machine, lock files are *not*
>  distributed locks).
> 
>  The api that we expose from oslo is typically accessed via the following:
> 
> oslo_concurrency.lockutils.synchronized(name, lock_file_prefix=None,
>  external=False, lock_path=None, semaphores=None, delay=0.01)
> 
>  or via its underlying library (that I extracted from oslo.concurrency
>  and have improved to add more usefulness) @
>  http://fasteners.readthedocs.org/
> 
>  The issue though for <> is that each of
>  these projects now typically has a large amount of lock files that exist
>  or have existed and no easy way to determine when those lock files can
>  be deleted (afaik no? periodic task exists in said projects to clean up
>  lock files, or to delete them when they are no longer in use...) so what
>  happens is bugs like https://bugs.launchpad.net/cinder/+bug/1432387
>  appear and there is no a simple solution to clean lock files up (since
>  oslo.concurrency is really not the right layer to know when a lock can
>  or can not be deleted, only the application knows that...)
> 
>  So then we get a few creative solutions like the following:
> 
>  - https://review.openstack.org/#/c/241663/
>  - https://review.openstack.org/#/c/239678/
>  - (and others?)
> 
>  So I wanted to ask the question, how are people involved in <  favorite openstack project>> cleaning up these files (are they at all?)
> 
>  Another idea that I have been proposing also is to use offset locks.
> 
>  This would allow for not creating X lock files, but create a *single*
>  lock file per project and use offsets into it as the way to lock. For
>  example nova could/would create a 1MB (or larger/smaller) *empty* file
>  for locks, that would allow for 1,048,576 locks to be used at the same
>  time, which honestly should be way more than enough, and then there
>  would not need to be any lock cleanup at all... Is there any reason this
>  wasn't initially done back way when this lock file code was created?
>  (https://github.com/harlowja/fasteners/pull/10 adds this functionality
>  to the underlying library if people want to look it over)
> >>>
> >>> I think the main reason was that even with a million locks available,
> >>> you'd have to find a way to hash the lock names to offsets in the file,
> >>> and a million isn't a very large collision space for that.  Having two
> >>> differently named locks that hashed to the same offset would lead to
> >>> incredibly confusing bugs.
> >>>
> >>> We could switch to requiring the projects to provide the offsets instead
> >>> of hashing a string value, but that's just pushing the collision problem
> >>> off onto every project that uses us.
> >>>
> >>> So that's the problem as I understand it, but where does that leave us
> >>> for solutions?  First, there's
> >>> https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L151
> >>> which allows consumers to delete lock files when they're done with them.
> >>>  Of course, in that case the onus is on the caller to make sure the lock
> >>> couldn't possibly be in use anymore.
> >>>
> >>> Second, is this actually a problem?  Modern filesystems have absurdly
> >>> large limits on the number of files in a directory, so it's highly
> >>> unlikely we would ever exhaust that, and we're creating all zero byte
> >>> files so there shouldn't be a significant space impact either.  In the
> >>> past I believe our recommendation has been to simply create a cleanup
> >>> job that runs on boot, before any of the OpenStack services start, that
> >>> deletes all of the lock files.  At that point you know it's safe to
> >>> delete them, and it prevents your lock file directory from growing 
> >>> forever.
> >>
> >> Not that high - ext3 

Re: [openstack-dev] Role in Congress

2015-11-30 Thread Adam Young

On 11/28/2015 05:03 AM, zhangyali (D) wrote:


Hi Tim and All,

I remember there is a topic named “role assignment for service users” 
in the Tokyo Summit. Since I have not heard any message of this topic. 
Does anyone could contribute some information for me? I think it is 
vital for my design of Congress UI in horizon. Thanks a lot!




There was a fgew places where service users needed "admin" which we need 
to root out.



What else are you thinking about?



Best Regards,

Yali



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread hao wang
Thanks very much, Anita, very usefully information, I will check it.

2015-11-30 23:36 GMT+08:00 Anita Kuno :
> On 11/29/2015 02:02 PM, Sean McGinnis wrote:
>> On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
>>> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
 Hi guys,

 I notice nova have a clarification of project scope:
 http://docs.openstack.org/developer/nova/project_scope.html

 I want to find cinder's, but failed,  do you know where to find it?

 It's important to let developers know what feature should be
 introduced into cinder and what shouldn't.

 BR
 Wang Hao
>>>
>>> I believe Nova team needed to formalize the scop to have an explanation
>>> for all the "this doesn't belong in Nova" comments on feature requests.
>>> Does Cinder suffer from similar problems? From my perspective it's not
>>> critically needed.
>>
>> I agree. I haven't seen a need for something like that with Cinder. Wang
>> Hao, is there a reason you feel you need that?
>>
>
> For reference here is the Cinder mission statement:
> http://git.openstack.org/cgit/openstack/governance/tree/reference/projects.yaml#n273
>
> All projects listed in the governance repository reference/projects.yaml
> have a mission statement, I do encourage folks thinking about starting a
> project to look at the mission statements here first as there may
> already be an effort ongoing with which you can align your work.
>
> Thanks Wang Hao,
> Anita.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [cinder]Do we have project scope for cinder?

2015-11-30 Thread hao wang
Thanks Xing Yang, I have noticed this spec, it's glad to see you to
start this work.

2015-11-30 23:10 GMT+08:00 yang, xing :
> Hi Wang Hao,
>
> Here¹s a Cinder spec in review on replicating a group of volumes:
>
> https://review.openstack.org/#/c/229722/
>
> It is a not an easy problem to solve.  Not that we should rush on this
> problem, but we should start thinking about how to solve this as some
> backends can only replicate a CG or a pool of volumes.
>
> Thanks,
> Xing
>
>
>
> On 11/30/15, 4:51 AM, "hao wang"  wrote:
>
>>Hi, Duncan
>>
>>2015-11-30 15:54 GMT+08:00 Duncan Thomas :
>>> Hi WangHao
>>>
>>> This was quite thoroughly discussed during the early discussions on
>>> replication. The general statement was 'not yet'. Getting any kind of
>>> workable replication API has proven to be very, very difficult to get
>>>right
>>> - we won't know for another full cycle whether we've actually gotten it
>>> somewhere near right, as operators start to deploy it. Piling more
>>>feature
>>> in the replication API before a) it has been used by operators and b)
>>> storage vendors have implemented what we already has would IMO be a
>>>mistake.
>>
>>I agree with you, in my mind, using replication what we have is first
>>thing we should done,
>>improve it much better is second thing, and then we will add another
>>new features
>>one by one stably.
>>
>>> None of this means that more DR interfaces don't belong in cinder, just
>>>that
>>> getting them right, getting them universal and getting them useful is
>>>quite
>>> a hard problem, and not one we should be in a rush to solve.
>>>Particularly as
>>> DR and replication is still a niche area of cinder, and we still have
>>>major
>>> issues in our basic functionality.
>>
>>Yes, this will convince me about DR in Cinder, very clearly, thanks.
>>>
>>> On 30 November 2015 at 03:45, hao wang  wrote:

 Sean and Michal,

 In fact, there is a reason that I ask this question. Recently I have a
 confusion about if cinder should provide the ability of Disaster
 Recovery to storage resources, like volume. I mean we have volume
 replication v1, but for DR, specially DR between two independent
 OpenStack sites(production and DR site), I feel we still need more
 features to support it, for example consistency group for replication,
 etc. I'm not sure if those features belong in Cinder or some new
 project for DR.

 BR
 WangHao

 2015-11-30 3:02 GMT+08:00 Sean McGinnis :
 > On Sun, Nov 29, 2015 at 11:44:19AM +, Dulko, Michal wrote:
 >> On Sat, 2015-11-28 at 10:56 +0800, hao wang wrote:
 >> > Hi guys,
 >> >
 >> > I notice nova have a clarification of project scope:
 >> > http://docs.openstack.org/developer/nova/project_scope.html
 >> >
 >> > I want to find cinder's, but failed,  do you know where to find
it?
 >> >
 >> > It's important to let developers know what feature should be
 >> > introduced into cinder and what shouldn't.
 >> >
 >> > BR
 >> > Wang Hao
 >>
 >> I believe Nova team needed to formalize the scop to have an
explanation
 >> for all the "this doesn't belong in Nova" comments on feature
requests.
 >> Does Cinder suffer from similar problems? From my perspective it's
not
 >> critically needed.
 >
 > I agree. I haven't seen a need for something like that with Cinder.
Wang
 > Hao, is there a reason you feel you need that?
 >
 >
 >
 >

__
 > OpenStack Development Mailing List (not for usage questions)
 > Unsubscribe:
 > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> --
>>> --
>>> Duncan Thomas
>>>
>>>
>>>_
>>>_
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>__
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> 

[openstack-dev] CentOS7 Merging Plan

2015-11-30 Thread Dmitry Teselkin
Hello,

We're almost got green BVT on custom CentOS7 ISO and it seems that it's
the time to discuss the plan how this feature could be merged.

This is not the only one feature that is in a queue. Unfortunately,
almost any other feature will be broken if merged after CentOS7, so it
was decided to merge our changes last.

This is not an official announcement, rather a notification letter to
start a discussion and find any objections.

So the plan is:

* merge all features that are going to be merged before Thusday, Dec 3
* call for merge freeze starting at Dec 3, due Dec 7
* rebase all CentOS7-related pathsets and resolve any conflicts with
  merged code (Dec 3)
* build custom ISO, pass BVT (and other tests) (Dec 3)
* merge all CentOS7-related patchsets at once (Dec 4)
* build an ISO and pass BVT again (Dec 4)
* run additional test during weekend (Dec 5, 6) to be sure that ISO
  good enough

According to this plan on Monday, Dec 7 we should either get CentOS7
based ISO, or revert all incompatible changes.

-- 
Thanks,
Dmitry Teselkin


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tripleo][ironic][heat] Adding back the tripleo check job

2015-11-30 Thread Zane Bitter

On 30/11/15 12:51, Ruby Loo wrote:



On 30 November 2015 at 10:19, Derek Higgins > wrote:

Hi All,

 A few months tripleo switch from its devtest based CI to one
that was based on instack. Before doing this we anticipated
disruption in the ci jobs and removed them from non tripleo projects.

 We'd like to investigate adding it back to heat and ironic as
these are the two projects where we find our ci provides the most
value. But we can only do this if the results from the job are
treated as voting.


What does this mean? That the tripleo job could vote and do a -1 and
block ironic's gate?


 In the past most of the non tripleo projects tended to ignore
the results from the tripleo job as it wasn't unusual for the job to
broken for days at a time. The thing is, ignoring the results of the
job is the reason (the majority of the time) it was broken in the
first place.
 To decrease the number of breakages we are now no longer
running master code for everything (for the non tripleo projects we
bump the versions we use periodically if they are working). I
believe with this model the CI jobs we run have become a lot more
reliable, there are still breakages but far less frequently.

What I proposing is we add at least one of our tripleo jobs back to
both heat and ironic (and other projects associated with them e.g.
clients, ironicinspector etc..), tripleo will switch to running
latest master of those repositories and the cores approving on those
projects should wait for a passing CI jobs before hitting approve.
So how do people feel about doing this? can we give it a go? A
couple of people have already expressed an interest in doing this
but I'd like to make sure were all in agreement before switching it on.

This seems to indicate that the tripleo jobs are non-voting, or at least
won't block the gate -- so I'm fine with adding tripleo jobs to ironic.
But if you want cores to wait/make sure they pass, then shouldn't they
be voting? (Guess I'm a bit confused.)


+1

I don't think it hurts to turn it on, but tbh I'm uncomfortable with the 
mental overhead of a non-voting job that I have to manually treat as a 
voting job. If it's stable enough to make it a voting job, I'd prefer we 
just make it voting. And if it's not then I'd like to see it be made 
stable enough to be a voting job and then make it voting.


- ZB

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [ironic] Third Party CI Deadlines for Mitaka and N

2015-11-30 Thread Mike Perez
On October 28th 2015 at the Ironic Third Party CI summit session [1], there was
consensus by the Ironic core and participating vendors that the set of
deadlines will be:

* Mitaka-2ː Driver teams will have registered their intent to run CI by creating
system accounts and identifying a point of contact for their CI team in the
Third party CI wiki [2].
* Mitaka Feature Freezeː All driver systems show the ability to receive events
and post comments in the sandbox.
* N release feature freezeː Per patch testing and posting comments.

There are requirements set for OpenStack Third Party CI's [3]. In addition
Ironic third party CI's must:

1) Test all drivers your company has integrated in Ironic.

For example, if your company has two drivers in Ironic, you would need to have
a CI that tests against the two and reports the results for each, for every
Ironic upstream patch. The tests come from a Devstack Gate job template [4], in
which you just need to switch the "deploy_driver" to your driver.

To get started, read OpenStack's third party testing documentation [5]. There
are efforts by OpenStack Infra to allow others to run third party CI similar to
the OpenStack upstream CI using Puppet [6] and instruction are available [7].
Don't forget to register your CI in the wiki [2], there is no need to announce
about it on any mailing list.

OpenStack Infra also provides third party CI help via meetings [8], and the
Ironic team has designated people to answer questions with setting up a third
party CI in the #openstack-ironic room [9].

If a solution does not have a CI watching for events and posting comments to
the sandbox [10] by the Mitaka feature freeze, it'll be assumed the driver is
not active, and can be removed from the Ironic repository as of the Mitaka
release.

If a solution is not being tested in a CI system and reporting to OpenStack
gerrit Ironic patches by the deadline of the N release feature freeze, an
Ironic driver could be removed from the Ironic repository. Without a CI system,
Ironic core is unable to verify your driver works in the N release of Ironic.

If there is something not clear about this email, please email me *directly*
with your question. You can also reach me as thingee on Freenode IRC in the
#openstack-ironic channel. Again I want you all to be successful in this, and
take advantage of this testing you will have with your product. Please
communicate with me and reach out to the team for help.

[1] - https://etherpad.openstack.org/p/summit-mitaka-ironic-third-party-ci
[2] - https://wiki.openstack.org/wiki/ThirdPartySystems
[3] - 
http://docs.openstack.org/infra/system-config/third_party.html#requirements
[4] - 
https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/devstack-gate.yaml#L961
[5] - http://docs.openstack.org/infra/system-config/third_party.html
[6] - https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/
[7] - 
https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/README.md
[8] - https://wiki.openstack.org/wiki/Meetings/ThirdParty
[9] - https://wiki.openstack.org/wiki/Ironic/Testing#Questions
[10] - https://review.openstack.org/#/q/project:+openstack-dev/sandbox,n,z

-- 
Mike Perez

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Evolving the stadium concept

2015-11-30 Thread Doug Wiegley

> On Nov 30, 2015, at 5:56 PM, Armando M.  wrote:
> 
> Hi folks,
> 
> The stadium concept was introduced more or less formally since April of this 
> year. At the time it was introduced (see [1]), the list of deliverables 
> included neutron, client, specs and *-aas services. As you may be well aware, 
> 6+ months is a long time in the OpenStack world, and lots of things happened 
> since then. The list has grown to [2]. 
> 
> When I think about what 'deliverables' are, I am inclined to think that all 
> of the projects that are part of the list will have to behave and follow the 
> same rules, provided that there is flexibility given by the tags. However, 
> reality has proven us that rules are somewhat difficult to follow and 
> enforce, and some boundaries may be too strict for some initiatives to comply 
> with. This is especially true if we go from a handful of projects that we had 
> when this had started to the nearly the two dozens we have now.
> As a result, there is quite an effort imposed on the PTL, the various 
> liaisons (release, infra, docs, testing, etc) and the core team to help 
> manage the existing relationships and to ensure that the picture stays 
> coherent over time. Sometimes the decision of being part of this list is even 
> presented before one can see any code, and that defeats the whole point of 
> the deliverable association. I have experienced first hand that this has 
> become a burden, and I fear that the stadium might be an extra layer of 
> governance/complexity that could even interfere with the existing 
> responsibilities of the TC and of OpenStack infra.
> 
> So my question is: would revisiting/clarifying the concept be due after some 
> time we have seen it in action? I would like to think so. To be fair, I am 
> not sure what the right answer is, but I know for a fact that some iterations 
> are in order, and I like to make a proposal:
> 
> I would like to suggest that we evolve the structure of the Neutron 
> governance, so that most of the deliverables that are now part of the Neutron 
> stadium become standalone projects that are entirely self-governed (they have 
> their own core/release teams, etc). In order to denote the initiatives that 
> are related to Neutron I would like to present two new tags that projects can 
> choose to label themselves with:
> 
Interesting proposal, and I’m just thinking out loud here. I’m generally in 
favor of separating the governance as we separate the dependencies, just 
because at some point what we’re doing doesn’t scale. To provide a little 
context, there are two points worth keeping in mind:

- The neutron stadium actually slightly pre-dates the big tent, and works 
around some earlier governance friction. So it may make less sense now in light 
of those changes.

- Many of the neutron subprojects *MUST RELEASE IN LOCKSTEP WITH NEUTRON* to be 
useful. These items are less useful to be considered standalone, as they need 
general oversight, co-gating, and such, to stay sane. As we break the massive 
coupling that exists, this point will get less and less relevant. 

- I think that part of the initial intent was that these small subprojects 
would have their own core teams, but be able to take advantage of the 
infrastructure that exists around neutron as a whole (specs, release team, 
stable team, co-gates, mentors).

For your proposal, are you suggesting:

1. That these projects are fully separate, with their own PTLs and everything, 
and just have tags that imply their neutron dependency?  OR,
2. That they stay stadium projects, but we use tags to differentiate them? Many 
already have different core teams and their own specs process.

Are there particular projects that add more overhead? Does your proposal make 
it easier to get code to our user base? Does it add a bunch of makework to fit 
into a new model (same question, really). Is the PTL overhead too high 
currently? Is the shed pink or blue?  :-)

Thanks,
doug




> 
> 'is-neutron-subsystem': this means that the project provides networking 
> services by implementing an integral part (or parts) of an end-to-end neutron 
> system. Examples are: a service plugin, an ML2 mech driver, a monolithic 
> plugin, an agent etc. It's something an admin has to use in order to deploy 
> Neutron in a certain configuration.
> 'use-neutron-system': this means that the project provides networking 
> services by using a pre-deployed end-to-end neutron system as is. No 
> modifications whatsoever.
> 
> As a result, there is no oversight by the Neutron core team, the PTL or 
> liasons, but that should not stop people from being involved if they choose 
> to. We would not lose the important piece of information which is the 
> association to Neutron, and at the same time that would relieve some of us 
> from the onus of dealing with initiatives for which we lack enough context 
> and ability of providing effective guidance.
> 
> In the process, the core team 

Re: [openstack-dev] [Neutron] Evolving the stadium concept

2015-11-30 Thread Armando M.
On 30 November 2015 at 20:47, Doug Wiegley 
wrote:

>
> On Nov 30, 2015, at 5:56 PM, Armando M.  wrote:
>
> Hi folks,
>
> The stadium concept was introduced more or less formally since April of
> this year. At the time it was introduced (see [1]), the list of
> deliverables included neutron, client, specs and *-aas services. As you may
> be well aware, 6+ months is a long time in the OpenStack world, and lots of
> things happened since then. The list has grown to [2].
>
> When I think about what 'deliverables' are, I am inclined to think that
> all of the projects that are part of the list will have to behave and
> follow the same rules, provided that there is flexibility given by the
> tags. However, reality has proven us that rules are somewhat difficult to
> follow and enforce, and some boundaries may be too strict for some
> initiatives to comply with. This is especially true if we go from a handful
> of projects that we had when this had started to the nearly the two dozens
> we have now.
>
> As a result, there is quite an effort imposed on the PTL, the various
> liaisons (release, infra, docs, testing, etc) and the core team to help
> manage the existing relationships and to ensure that the picture stays
> coherent over time. Sometimes the decision of being part of this list is
> even presented before one can see any code, and that defeats the whole
> point of the deliverable association. I have experienced first hand that
> this has become a burden, and I fear that the stadium might be an extra
> layer of governance/complexity that could even interfere with the existing
> responsibilities of the TC and of OpenStack infra.
>
> So my question is: would revisiting/clarifying the concept be due after
> some time we have seen it in action? I would like to think so. To be
> fair, I am not sure what the right answer is, but I know for a fact that
> some iterations are in order, and I like to make a proposal:
>
> I would like to suggest that we evolve the structure of the Neutron
> governance, so that most of the deliverables that are now part of the
> Neutron stadium become standalone projects that are entirely
> self-governed (they have their own core/release teams, etc). In order to
> denote the initiatives that are related to Neutron I would like to present
> two new tags that projects can choose to label themselves with:
>
> Interesting proposal, and I’m just thinking out loud here. I’m generally
> in favor of separating the governance as we separate the dependencies, just
> because at some point what we’re doing doesn’t scale. To provide a little
> context, there are two points worth keeping in mind:
>
> - The neutron stadium actually slightly pre-dates the big tent, and works
> around some earlier governance friction. So it may make less sense now in
> light of those changes.
>

If my memory doesn't fail me, the first time I recall of talks about the
big tent is September 2014, in this email thread:

http://lists.openstack.org/pipermail/openstack-dev/2014-September/046437.html

So, no I don't think we were a novelty :)


>
> - Many of the neutron subprojects *MUST RELEASE IN LOCKSTEP WITH NEUTRON*
> to be useful. These items are less useful to be considered standalone, as
> they need general oversight, co-gating, and such, to stay sane. As we break
> the massive coupling that exists, this point will get less and less
> relevant.
>

I don't think that requires the oversight of a single individual, but
you're right that this point will fade away over time.


>
> - I think that part of the initial intent was that these small subprojects
> would have their own core teams, but be able to take advantage of the
> infrastructure that exists around neutron as a whole (specs, release team,
> stable team, co-gates, mentors).
>
> For your proposal, are you suggesting:
>
> 1. That these projects are fully separate, with their own PTLs and
> everything, and just have tags that imply their neutron dependency?  OR,
>

My point is: the project decides what's best, I don't have to have an
opinion :)

If they want to signal a relationship with Neutron they can do so by
choosing one of the two tags being proposed.


> 2. That they stay stadium projects, but we use tags to differentiate them?
> Many already have different core teams and their own specs process.
>

Must there be a place where we are together that is not OpenStack already?
And what would that together mean exactly? That's the conundrum I am trying
to move away from


>
> Are there particular projects that add more overhead? Does your proposal
> make it easier to get code to our user base? Does it add a bunch of
> makework to fit into a new model (same question, really). Is the PTL
> overhead too high currently? Is the shed pink or blue?  :-)
>

Not sure what you're asking: this isn't just about overhead to a single or
a small group of people. It's about arranging ourselves the best way
possible in order 

Re: [openstack-dev] [all][glance] Add Sabari Kumar Murugesan <smuruge...@vmware.com>

2015-11-30 Thread Nikhil Komawar
HI,

Great to see him (Sabari) finally being proposed after new cycle
settling wait. I agree he has always done good work and shows great
awareness and expertise on the store (data transfer) fronts. Definitely
going to be a great core.

(For context) Like my previous shout out [1] to (Sabari &) Abhishek [2],
I would like to see both of them in the core team. Abhishek has shown
good awareness for cross project stuff and has provided some really good
reviews (gotchas, security oops, etc), filed and resolved important
bugs, helped the team with knowledge on scarcely used glance (and
import) code.

I think Glance team couldn't hurt with some extra reviewers especially
across the timezones with Fei Long being more busy with his other
commitments. So, firstly apologies for late reply and secondly I hope to
see some more (re-)voting on this adjustment.

[1]
http://eavesdrop.openstack.org/meetings/glance/2015/glance.2015-10-01-14.01.log.html#l-70
[2] https://launchpad.net/~abhishek-kekane


On 11/25/15 6:53 AM, Kuvaja, Erno wrote:
>> -Original Message-
>> From: Flavio Percoco [mailto:fla...@redhat.com]
>> Sent: Monday, November 23, 2015 8:21 PM
>> To: openstack-dev@lists.openstack.org
>> Cc: Sabari Kumar Murugesan
>> Subject: [openstack-dev] [all][glance] Add Sabari Kumar Murugesan
>> 
>>
>> Greetings,
>>
>> I'd like to propose adding Sabari Kumar Murugesan to the glance-core team.
>> Sabari has been contributing for quite a bit to the project with great 
>> reviews
>> and he's also been providing great feedback in matters related to the design
>> of the service, libraries and other areas of the team.
>>
>> I believe he'd be a great addition to the glance-core team as he has
>> demonstrated a good knowledge of the code, service and project's priorities.
>>
>> If Sabari accepts to join and there are no objections from other members of
>> the community, I'll proceed to add Sabari to the team in a week from now.
>>
>> Thanks,
>> Flavio
>>
>> --
>> @flaper87
>> Flavio Percoco
> +2
> - Erno
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 

Thanks,
Nikhil



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [keystone]Different projects authentication strategy

2015-11-30 Thread 1021710773
Every Developers,


Hello. I here would like to ask some questions about policy rules.
Now the policy rules of openstack in keystone and other projects are set in 
policy.json, in other words, the policy rules are equal
to each projects. And the common ways to enforce are in decorative function 
like protected(). And in keystone project, it manage the users, projects,  
roles and other resources. Now, some particular projects(tenants) may have its 
own enforce rules, not just like the policy.json, and in that ways, could we 
update the usual decorative function of enforce to realize the authentification 
of projects? And now, the policy model appears in keystone project. Could we 
use it to create association between projects and policy? 
Hope to hear from you. Thanks!




Weiwei Yang

yangwei...@cmss.chinamobile.com__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [openstack-infra] Project approved specs

2015-11-30 Thread Steve Martinelli

you probably want to propose a change to this file:
https://github.com/openstack-infra/project-config/blob/master/specs/specs.yaml

Thanks,

Steve Martinelli
OpenStack Keystone Project Team Lead



From:   "GROSZ, Maty (Maty)" 
To: "OpenStack Development Mailing List (not for usage questions)"

Date:   2015/12/01 01:21 AM
Subject:[openstack-dev] [openstack-infra] Project approved specs




Hey,

All vitrage specs are written and kept in the vitrage-specs repository.
According to the developers guide, all approved specs appear here:
http://specs.openstack.org/
How do the approved specs get to appear in the link above?
Is there a writing convention that need to be followed? Any configuration?

Thanks

Maty
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Evolving the stadium concept

2015-11-30 Thread Brandon Logan
On Mon, 2015-11-30 at 23:11 -0500, Russell Bryant wrote:
> Some additional context: there are a few proposals for additional git
> repositories for Neutron that have been put on hold while we sort this out.
> 
> Add networking-bagpipe:
>   https://review.openstack.org/#/c/244736/
> 
> Add the Astara driver:
>   https://review.openstack.org/#/c/230699/
> 
> Add tap-as-a-service:
>   https://review.openstack.org/#/c/229869/
> 
> On 11/30/2015 07:56 PM, Armando M. wrote:
> > I would like to suggest that we evolve the structure of the Neutron
> > governance, so that most of the deliverables that are now part of the
> > Neutron stadium become standalone projects that are entirely
> > self-governed (they have their own core/release teams, etc). In order to
> > denote the initiatives that are related to Neutron I would like to
> > present two new tags that projects can choose to label themselves with:
> > 
> >   * 'is-neutron-subsystem': this means that the project provides
> > networking services by implementing an integral part (or parts) of
> > an end-to-end neutron system. Examples are: a service plugin, an ML2
> > mech driver, a monolithic plugin, an agent etc. It's something an
> > admin has to use in order to deploy Neutron in a certain configuration.
> >   * 'use-neutron-system': this means that the project provides
> > networking services by using a pre-deployed end-to-end neutron
> > system as is. No modifications whatsoever.
> 
> I just want to clarify the proposal.  IIUC, you propose splitting most
> of what is currently separately deliverables of the Neutron team and
> making them separate projects in terms of OpenStack governance.  When I
> originally proposed including networking-ovn under Neutron (and more
> generally, making room for all drivers to be included), making them
> separate projects was one of the options on the table, but it didn't
> seem best at the time.  For reference, that thread was here:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2015-April/062310.html
> 
> When I was originally proposing this, I was only thinking about Neutron
> drivers, the stuff that connects Neutron to some other system to make
> Neutron do something.  The list has grown to include other things, as well.
> 
> I'm not sure where you propose the line to be, but for the sake of
> discussion, let's assume every deliverable in the governance definition
> for Neutron is under consideration for being split out with the
> exception of neutron, neutron-specs, and python-neutronclient.  The
> remaining deliverables are:
> 
> dragonflow:
> kuryr:
> networking-ale-omniswitch:
> networking-arista:
> networking-bgpvpn:
> networking-calico:
> networking-cisco:
> networking-fortinet:
> networking-hpe:
> networking-hyperv:
> networking-infoblox:
> networking-fujitsu:
> networking-l2gw:
> networking-lenovo:
> networking-midonet:
> networking-odl:
> networking-ofagent:
> networking-onos:
> networking-ovn:
> networking-plumgrid:
> networking-powervm:
> networking-sfc:
> networking-vsphere:
> octavia:
> python-neutron-pd-driver:
> vmware-nsx:
> 
> I think it's helpful to break these into categories, because the answer
> may be different for each group.  Here's my attempt at breaking this
> list into some categories:
> 
> 1) A consumer of Neutron
> 
> kuryr
> 
> IIUC, kuryr is a consumer of Neutron.  Its interaction with Neutron is
> via using Neutron's REST APIs.  You could think of kuryr's use of
> Neutron as architecturally similar to how Nova uses Neutron.
> 
> I think this project makes a ton of sense to become independent.
> 
> 2) Implementation of a networking technology
> 
> dragonflow
> 
> The dragonflow repo includes a couple of things.  It includes dragonflow
> itself, and the Neutron driver to connect to it.  Using Astara as an
> example to follow, dragonflow itself could be an independent project.
> 
> Following that, the built-in ML2/ovs or ML2/lb control plane could be
> separate, too, though that's much more painful and complex in practice.
> 
> octavia
> 
> Octavia also seems to fall into this category, just for LBaaS.  It's not
> just a driver, it's a LBaaS service VM orchestrator (which is in part
> what Astara is, too).

Actually I would put Octavia in #1 as it only interacts with neutron
through its REST API.  There is a neutron-lbaas octavia driver that
simply just calls the Octavia REST API, but it lives in the
neutron-lbaas tree.  Octavia is standalone and consumes all openstack
services through their REST APIs.

> 
> It seems reasonable to propose these as independent projects.
> 
> 3) New APIs
> 
> There are some repos that are implementing new REST APIs for Neutron.
> They're independent enough to need their own driver layer, but coupled
> with Neutron enough to still need to run inside of Neutron as they can't
> do everything they need to do by only 

Re: [openstack-dev] [all][glance] Add Sabari Kumar Murugesan <smuruge...@vmware.com>

2015-11-30 Thread Bhandaru, Malini K
+1 on Sabari!  :-)

-Original Message-
From: Flavio Percoco [mailto:fla...@redhat.com] 
Sent: Monday, November 23, 2015 12:21 PM
To: openstack-dev@lists.openstack.org
Cc: Sabari Kumar Murugesan 
Subject: [openstack-dev] [all][glance] Add Sabari Kumar Murugesan 


Greetings,

I'd like to propose adding Sabari Kumar Murugesan to the glance-core team. 
Sabari has been contributing for quite a bit to the project with great reviews 
and he's also been providing great feedback in matters related to the design of 
the service, libraries and other areas of the team.

I believe he'd be a great addition to the glance-core team as he has 
demonstrated a good knowledge of the code, service and project's priorities.

If Sabari accepts to join and there are no objections from other members of the 
community, I'll proceed to add Sabari to the team in a week from now.

Thanks,
Flavio

--
@flaper87
Flavio Percoco
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Evolving the stadium concept

2015-11-30 Thread Doug Wiegley

> On Nov 30, 2015, at 9:11 PM, Russell Bryant  wrote:
> 
> Some additional context: there are a few proposals for additional git
> repositories for Neutron that have been put on hold while we sort this out.
> 
> Add networking-bagpipe:
>  https://review.openstack.org/#/c/244736/
> 
> Add the Astara driver:
>  https://review.openstack.org/#/c/230699/
> 
> Add tap-as-a-service:
>  https://review.openstack.org/#/c/229869/
> 
> On 11/30/2015 07:56 PM, Armando M. wrote:
>> I would like to suggest that we evolve the structure of the Neutron
>> governance, so that most of the deliverables that are now part of the
>> Neutron stadium become standalone projects that are entirely
>> self-governed (they have their own core/release teams, etc). In order to
>> denote the initiatives that are related to Neutron I would like to
>> present two new tags that projects can choose to label themselves with:
>> 
>>  * 'is-neutron-subsystem': this means that the project provides
>>networking services by implementing an integral part (or parts) of
>>an end-to-end neutron system. Examples are: a service plugin, an ML2
>>mech driver, a monolithic plugin, an agent etc. It's something an
>>admin has to use in order to deploy Neutron in a certain configuration.
>>  * 'use-neutron-system': this means that the project provides
>>networking services by using a pre-deployed end-to-end neutron
>>system as is. No modifications whatsoever.
> 
> I just want to clarify the proposal.  IIUC, you propose splitting most
> of what is currently separately deliverables of the Neutron team and
> making them separate projects in terms of OpenStack governance.  When I
> originally proposed including networking-ovn under Neutron (and more
> generally, making room for all drivers to be included), making them
> separate projects was one of the options on the table, but it didn't
> seem best at the time.  For reference, that thread was here:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2015-April/062310.html
> 
> When I was originally proposing this, I was only thinking about Neutron
> drivers, the stuff that connects Neutron to some other system to make
> Neutron do something.  The list has grown to include other things, as well.
> 
> I'm not sure where you propose the line to be, but for the sake of
> discussion, let's assume every deliverable in the governance definition
> for Neutron is under consideration for being split out with the
> exception of neutron, neutron-specs, and python-neutronclient.  The
> remaining deliverables are:
> 
>dragonflow:
>kuryr:
>networking-ale-omniswitch:
>networking-arista:
>networking-bgpvpn:
>networking-calico:
>networking-cisco:
>networking-fortinet:
>networking-hpe:
>networking-hyperv:
>networking-infoblox:
>networking-fujitsu:
>networking-l2gw:
>networking-lenovo:
>networking-midonet:
>networking-odl:
>networking-ofagent:
>networking-onos:
>networking-ovn:
>networking-plumgrid:
>networking-powervm:
>networking-sfc:
>networking-vsphere:
>octavia:
>python-neutron-pd-driver:
>vmware-nsx:
> 
> I think it's helpful to break these into categories, because the answer
> may be different for each group.  Here's my attempt at breaking this
> list into some categories:
> 
> 1) A consumer of Neutron
> 
>kuryr
> 
> IIUC, kuryr is a consumer of Neutron.  Its interaction with Neutron is
> via using Neutron's REST APIs.  You could think of kuryr's use of
> Neutron as architecturally similar to how Nova uses Neutron.
> 
> I think this project makes a ton of sense to become independent.
> 
> 2) Implementation of a networking technology
> 
>dragonflow
> 
> The dragonflow repo includes a couple of things.  It includes dragonflow
> itself, and the Neutron driver to connect to it.  Using Astara as an
> example to follow, dragonflow itself could be an independent project.
> 
> Following that, the built-in ML2/ovs or ML2/lb control plane could be
> separate, too, though that's much more painful and complex in practice.
> 
>octavia
> 
> Octavia also seems to fall into this category, just for LBaaS.  It's not
> just a driver, it's a LBaaS service VM orchestrator (which is in part
> what Astara is, too).

From the perspective of our users, I tend to consider neutron-lbaas and octavia 
as a unit, technical distinctions aside.

> 
> It seems reasonable to propose these as independent projects.
> 
> 3) New APIs
> 
> There are some repos that are implementing new REST APIs for Neutron.
> They're independent enough to need their own driver layer, but coupled
> with Neutron enough to still need to run inside of Neutron as they can't
> do everything they need to do by only interfacing with Neutron REST APIs
> (today, at least).
> 
>networking-l2gw:
>networking-sfc:
> 
> Here things start to get less clear to me.  Unless the only interaction
> with Neutron is via its REST API, 

Re: [openstack-dev] [Neutron] Evolving the stadium concept

2015-11-30 Thread Doug Wiegley

> On Nov 30, 2015, at 10:15 PM, Armando M.  wrote:
> 
> 
> 
> On 30 November 2015 at 20:47, Doug Wiegley  > wrote:
> 
>> On Nov 30, 2015, at 5:56 PM, Armando M. > > wrote:
>> 
>> Hi folks,
>> 
>> The stadium concept was introduced more or less formally since April of this 
>> year. At the time it was introduced (see [1]), the list of deliverables 
>> included neutron, client, specs and *-aas services. As you may be well 
>> aware, 6+ months is a long time in the OpenStack world, and lots of things 
>> happened since then. The list has grown to [2]. 
>> 
>> When I think about what 'deliverables' are, I am inclined to think that all 
>> of the projects that are part of the list will have to behave and follow the 
>> same rules, provided that there is flexibility given by the tags. However, 
>> reality has proven us that rules are somewhat difficult to follow and 
>> enforce, and some boundaries may be too strict for some initiatives to 
>> comply with. This is especially true if we go from a handful of projects 
>> that we had when this had started to the nearly the two dozens we have now.
>> As a result, there is quite an effort imposed on the PTL, the various 
>> liaisons (release, infra, docs, testing, etc) and the core team to help 
>> manage the existing relationships and to ensure that the picture stays 
>> coherent over time. Sometimes the decision of being part of this list is 
>> even presented before one can see any code, and that defeats the whole point 
>> of the deliverable association. I have experienced first hand that this has 
>> become a burden, and I fear that the stadium might be an extra layer of 
>> governance/complexity that could even interfere with the existing 
>> responsibilities of the TC and of OpenStack infra.
>> 
>> So my question is: would revisiting/clarifying the concept be due after some 
>> time we have seen it in action? I would like to think so. To be fair, I am 
>> not sure what the right answer is, but I know for a fact that some 
>> iterations are in order, and I like to make a proposal:
>> 
>> I would like to suggest that we evolve the structure of the Neutron 
>> governance, so that most of the deliverables that are now part of the 
>> Neutron stadium become standalone projects that are entirely self-governed 
>> (they have their own core/release teams, etc). In order to denote the 
>> initiatives that are related to Neutron I would like to present two new tags 
>> that projects can choose to label themselves with:
>> 
> 
> Interesting proposal, and I’m just thinking out loud here. I’m generally in 
> favor of separating the governance as we separate the dependencies, just 
> because at some point what we’re doing doesn’t scale. To provide a little 
> context, there are two points worth keeping in mind:
> 
> - The neutron stadium actually slightly pre-dates the big tent, and works 
> around some earlier governance friction. So it may make less sense now in 
> light of those changes.
> 
> If my memory doesn't fail me, the first time I recall of talks about the big 
> tent is September 2014, in this email thread:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2014-September/046437.html 
> 
> 
> So, no I don't think we were a novelty :)
>  
> 
> - Many of the neutron subprojects *MUST RELEASE IN LOCKSTEP WITH NEUTRON* to 
> be useful. These items are less useful to be considered standalone, as they 
> need general oversight, co-gating, and such, to stay sane. As we break the 
> massive coupling that exists, this point will get less and less relevant. 
> 
> I don't think that requires the oversight of a single individual, but you're 
> right that this point will fade away over time.
>  
> 
> - I think that part of the initial intent was that these small subprojects 
> would have their own core teams, but be able to take advantage of the 
> infrastructure that exists around neutron as a whole (specs, release team, 
> stable team, co-gates, mentors).
> 
> For your proposal, are you suggesting:
> 
> 1. That these projects are fully separate, with their own PTLs and 
> everything, and just have tags that imply their neutron dependency?  OR,
> 
> My point is: the project decides what's best, I don't have to have an opinion 
> :)

I don’t think you *have* to have an opinion in either scheme. That sounds 
self-imposed. Delegate it.

> 
> If they want to signal a relationship with Neutron they can do so by choosing 
> one of the two tags being proposed.
>  
> 2. That they stay stadium projects, but we use tags to differentiate them? 
> Many already have different core teams and their own specs process.
> 
> Must there be a place where we are together that is not OpenStack already? 
> And what would that together mean exactly? That's the conundrum I am 

  1   2   >