date:20130709

On 10 July 2013 02:31, Monty Taylor  wrote:
> 
> 
> On 07/09/2013 10:55 AM, Joe Gordon wrote:
>> 
>> On Tue, Jul 9, 2013 at 1:28 PM, Kieran Spear > > wrote:
>> 
>>Hi all,
>> 
>>There's a review up to make Horizon pass H304 (no relative imports):
>> 
>>https://review.openstack.org/#/c/35664/
>> 
>>Old-style relative imports are dangerous, but I can't think of a
>>good enough reason to forbid new-style imports, e.g.:
>> 
>>from .views import IndexView
>> 
>>instead of
>> 
>>from openstack_dashboard.dashboards.project.images_and_snapshots.views \
>>  import IndexView
>> 
>>particularly in Horizon where nesting is deep and you otherwise end
>>up with many imports that won't fit into 80 characters. I think if
>>we're going to make the change above the benefits would need to
>>outweigh the effect on readability.
>> 
>>There's some discussion on the review already, but I'd like to hear
>>from a more general audience. Thoughts?
>> 
>> 
>> 
>> http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Imports
>> 
>> "Do not use relative names in imports. Even if the module is in the same
>> package, use the full package name. This helps prevent unintentionally
>> importing a package twice."
>> 
> 
> From pep8:
> 
> Relative imports for intra-package imports are highly discouraged.
> Always use the absolute package path for all imports. Even now that PEP
> 328 is fully implemented in Python 2.5, its style of explicit relative
> imports is actively discouraged; absolute imports are more portable and
> usually more readable.

In Horizon I think it's obvious that relative imports are more readable (and 
type-able) than the alternative. If portable is referring to moving code 
around, I think each style has portability benefits for both inter- and 
intra-package module moves, and there's a lot of overlap.

It sounds like this particular paragraph has been updated to account for PEP328 
without considering why new-style imports should be discouraged.

There was a thread on python-dev a few years ago about the same issue:

http://mail.python.org/pipermail/python-dev/2010-October/104476.html

On 10/5/2010 2:21 PM, Guido van Rossum wrote:
> On Tue, Oct 5, 2010 at 11:17 AM, Darren Dale  wrote:
>> The issue is implementing a PEP with nice support for relative
>> imports, and then documenting that it should never be used.
> 
> Isn't this mostly historical? Until the new relative-import syntax was
> implemented there were various problems with relative imports. The
> short-term solution was to recommend not using them. The long-term
> solution was to implement an unambiguous syntax. Now it is time to
> withdraw the anti-recommendation. Of course, without going overboard
> -- I still find them an acquired taste; but they have their place.

But nothing ever came of the bug:
http://bugs.python.org/issue10031



> 
> Monty

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [ml2] ML2 Sub-Team Meeting tomorrow

2013-07-09 Thread Andre Pech

Thanks Kyle,

I'm unfortunately going to be on a plane during tomorrow's meeting, so
wanted to send a quick update on the ML2 mechanism driver infrastructure (
https://blueprints.launchpad.net/neutron/+spec/ml2-mechanism-drivers).
Thanks to everyone for the latest rounds of review, I believe that I've
incorporated all of the feedback so far and appreciate people taking
another look. I'm available most of tomorrow so will try to be responsive
to comments so that we can hopefully get this merged and into H2.

Thanks
Andre

On Tue, Jul 9, 2013 at 7:20 PM, Kyle Mestery (kmestery)
wrote:

> Just a reminder, we have our weekly ML2 sub team meeting Wednesday at
> 1400UTC. The agenda is on the meeting page here:
>
> https://wiki.openstack.org/wiki/Meetings/ML2
>
> We have a number of ML2 related blueprints in review with a hope of
> getting them in for H2 tomorrow, we'll focus on those in the meeting
> tomorrow for the most part.
>
> Thanks!
> Kyle
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] HACKING and new-style relative imports

On 10 July 2013 00:55, Joe Gordon  wrote:
> 
> On Tue, Jul 9, 2013 at 1:28 PM, Kieran Spear  wrote:
>> 
>> Hi all,
>> 
>> There's a review up to make Horizon pass H304 (no relative imports):
>> 
>> https://review.openstack.org/#/c/35664/
>> 
>> Old-style relative imports are dangerous, but I can't think of a good
>> enough reason to forbid new-style imports, e.g.:
>> 
>> from .views import IndexView
>> 
>> instead of
>> 
>> from openstack_dashboard.dashboards.project.images_and_snapshots.views \
>>  import IndexView
>> 
>> particularly in Horizon where nesting is deep and you otherwise end up
>> with many imports that won't fit into 80 characters. I think if we're going
>> to make the change above the benefits would need to outweigh the effect on
>> readability.
>> 
>> There's some discussion on the review already, but I'd like to hear from a
>> more general audience. Thoughts?
> 
> 
> 
> http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Imports
> 
> "Do not use relative names in imports. Even if the module is in the same
> package, use the full package name. This helps prevent unintentionally
> importing a package twice."

Thanks. My understanding is that a module can only be imported twice if there 
are two different paths to it in your python path. This can happen with both 
relative and absolute imports, so I don't think H304 prevents that one.

> 
> 
>> 
>> 
>> Cheers,
>> Kieran
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] HACKING and new-style relative imports

On 09/07/2013, at 9:57 PM, Sean Dague  wrote:

> On 07/09/2013 07:28 AM, Kieran Spear wrote:
>> Hi all,
>> 
>> There's a review up to make Horizon pass H304 (no relative imports):
>> 
>> https://review.openstack.org/#/c/35664/
>> 
>> Old-style relative imports are dangerous, but I can't think of a good enough 
>> reason to forbid new-style imports, e.g.:
>> 
>> from .views import IndexView
>> 
>> instead of
>> 
>> from openstack_dashboard.dashboards.project.images_and_snapshots.views \
>>   import IndexView
> 
> However, if you decide to address H302, most of the over 80 character issues 
> in horizon are going to fit, as this would become:
> 
> from openstack_dashboard.dashboards.project.images_and_snapshots import views
> 
> (tacking in at 78 chars)
> 
> I'm sure there will still be a few which break the limit... but that will fix 
> a lot of them (from what I see in the review).

Agreed, that would work. But I'd like to consider H304 on its merits before we 
go working around it. There has also been some discussion about whether we want 
to enforce H302 in Horizon, particularly for classes.

> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [jenkins] [tempest] Trouble with a patch

2013-07-09 Thread Gareth

nice jobs!


On Wed, Jul 10, 2013 at 8:12 AM, Jeremy Stanley  wrote:

> On 2013-07-09 23:55:03 + (+), Kyle Mestery (kmestery) wrote:
> > Thanks for the note Jeremy! Can you shoot another reply on this
> > thread so those of us with patches failing can resubmit?
>
> Yep, now that https://review.openstack.org/36349 has merged you
> should be safe to recheck any proposed changes or reverify any
> approved changes which failed with that error.
> --
> Jeremy Stanley
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Gareth

*Cloud Computing, OpenStack, Fitness, Basketball*
*OpenStack contributor*
*Company: UnitedStack *
*My promise: if you find any spelling or grammar mistakes in my email from
Mar 1 2013, notify me *
*and I'll donate $1 or ￥1 to an open organization you specify.*
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] swift-bench 1.9.1-dev - AttributeError: Values instance has no attribute 'containers'

I recently downloaded swift 1.9.1-dev.

swift-bench gets the following error. What can I change to get this working 
sucessfully?

Thanks,

Tim



root@controller21:~/ssbench-0.2.16# 
python -c 'import swift; print swift.__version__'
1.9.1-dev
root@controller21:~/ssbench-0.2.16#

swift-bench -A http://localHost:8080/auth/v1.0 -K testing  -U test:tester -s 10 
-n 2 -g 1
swift-bench 2013-07-09 19:17:00,338 INFO Auth version: 1.0
Traceback (most recent call last):
  File "/usr/bin/swift-bench", line 149, in 
controller.run()
  File "/root/swift/swift/common/bench.py", line 372, in run
puts = BenchPUT(self.logger, self.conf, self.names)
  File "/root/swift/swift/common/bench.py", line 450, in __init__
self.containers = conf.containers
AttributeError: Values instance has no attribute 'containers'
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Campus Network Blueprint

On Jul 9, 2013, at 8:40 PM, Filipe Manco  wrote:
> 
> Hi Kyle
> 
> I already looked at the blueprints and I want to integrate my work with ML2, 
> mainly because I want users to able to keep the traditional networking models 
> working in parallel (and I think ML2 is the best way to do this comparing 
> with the meta plugin).
> 
> To be honest, the integration with Neutron and the ML2 was what took most of 
> the time when writing the blueprint, but although I talk about it on the 
> blueprint I still not sure about the best way to do it.
> 
> About the concept of segments I don't think they fit. From what I understand 
> a segment, in the context of ML2, represents part of the virtual network that 
> uses some technology to connect a set of VMs, so if I delete a segment those 
> VMs will loose connectivity. In the context of my blueprint a segment (or 
> cSegment as I called it) represents a domain where I can create virtual links 
> across a set of nodes. If I delete a cSegment that doesn't mean any VM will 
> loose connectivity, what will happen is that the network is remapped and the 
> traffic will cross another cSegment or another set of cSegments.
> 
I think what you've mentioned here is valid, yes, and is a slight deviation 
between the way segments in ML2 work on your cSegments. It looks like we would 
need to add your new constructs into the ML2 API to achieve what you're looking 
for.

> Something that I would like you (or other guys from ML2) to comment is how to 
> integrate new operations (beyond the ones related with the base data model) 
> in the ML2 interfaces. The MechanismDriver interface supports operations on 
> ports and networks, but as you can see I have new entities. My idea is that 
> there should be no changes to the original interfaces, because this are 
> specific to a type of network. What do you think?
> 
I'll have a look at this tomorrow and give you more detailed feedback. If you 
want, you can join us in the ML2 meeting at 1400UTV meeting on 
#openstack-meeting and I'll save some time at the end of the meeting to discuss 
your blueprint.

Thanks!
Kyle

> It would help me if you have the time to take a look at the blueprint and 
> comment on the ML2 parts.
> 
> Thanks for your help!
> 
> Filipe Manco
> http://about.me/fmanco
> 
> 
> 2013/7/9 Kyle Mestery (kmestery) 
> On Jul 9, 2013, at 11:18 AM, Filipe Manco  wrote:
> >
> > Hi all,
> >
> > Just published a blueprint for some work I'm starting right now. I would 
> > appreciate if someone can take a look and give some comments.
> >
> > https://blueprints.launchpad.net/neutron/+spec/campus-network
> >
> > Thank you
> >
> Hi Filipe:
> 
> At first glance, there appears to be a lot of overlap with ML2 here. Have you 
> looked at the ML2 blueprints, and the specific use case of multi-segment 
> networks? The ML2 team is working hard to close ML2 blueprints in H2/H3, 
> perhaps some of your ideas could be incorporated here?
> 
> Here is the ML2 Meeting page which has links to the blueprints we're tracking:
> 
> https://wiki.openstack.org/wiki/Meetings/ML2
> 
> Thanks,
> Kyle
> 
> >
> > Filipe Manco
> > http://about.me/fmanco
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] [ml2] ML2 Sub-Team Meeting tomorrow

Just a reminder, we have our weekly ML2 sub team meeting Wednesday at 1400UTC. 
The agenda is on the meeting page here:

https://wiki.openstack.org/wiki/Meetings/ML2

We have a number of ML2 related blueprints in review with a hope of getting 
them in for H2 tomorrow, we'll focus on those in the meeting tomorrow for the 
most part.

Thanks!
Kyle
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Service Type Framework implementation

Hi Mark


2013/7/9 Mark McClain :
>
> On Jul 9, 2013, at 5:37 PM, Nachi Ueno  wrote:
>>
>> We have two suboption for db api based solution
>>
>> Option4. REST API + DB with Preload with Conf
>> https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf14b7b30_00
>>
>> so IMO, we can drop  option3.
>>
>> I believe option4 is easy to implement.
>>
>
> I'm not onboard with option 4 either.  At the last summit, we talked about 
> making Neutron easier to deploy.  Using a database to sync configuration 
> state adds complexity.  Having some values in a configuration and others in 
> the database (even cached) is a recipe for a major headache.  For the 
> deployments running multiple instances of Neutron, they should be using Chef, 
> Chef, Salt, etc for managing their configs anyway.
>
> Using only configuration files (option 1) remains my preference.

"only configuration files (option 1)"  is also acceptable for me.
However, the headache continues even if we choose option1, because
relation with service type
and service resources are in the DB.

Note that we still need to provide way to add or remove service types.

Option1-1)
   Allow to create new relation if it appears in the conf.
   Remove the relation if it is disappears from conf.

   IMO, This will fall on same problem of current implementation
   
https://docs.google.com/a/ntti3.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf0f4e2a2_1136

Option1-2) Provide admin rest api for enable/disable service types
Allow to create new relation if it is enabled by API
 Remove the relation if it disabled by API

This is my preference. And IMO, this is same as option4.

Best
Nachi




> mark
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

2013-07-09 Thread Robert Collins

On 10 July 2013 11:00, Jay Pipes  wrote:

> >   I am not up to speed on"InnoDB's gap locking behavior" but it is
>
>> not something I would expect to be a problem in Postgresql.
>>
>
> InnoDB and PostgreSQL behave in very different manners regarding locking
> and transaction isolation, even though they both implement a version of
> MVCC. In InnoDB's case, its implementation of MVCC is optimized more for
> storage space (it allows using a series of log records to reconstruct or
> undo a record to a particular "version" of the record) vs. PostgreSQL,
> which stores every version of every record in its data space.
>

PostgreSQL only keeps enough versions live to ensure the oldest transaction
can still read all the versions that were live at it's start: once the
transaction is closed any obsolete version kept alive by it can be gc'd or
overwritten. I think the distinction you are drawing is that InnoDB only
stores deltas between record versions - an interesting tweak. I guess that
TOAST mitigates the temporary storage overhead: as a particular TOAST value
is immutable, there's no need to copy that data when updating a different
value in a row. That won't help with 'update one int in each of a billion
rows' case though.

> AFAIK, PostgreSQL won't issue gap locks like InnoDB because it will simply
> write a new version of the records -- a version that marks the record as
> deleted -- to its data files, instead of gap locking to ensure that rows
> affected by a DELETE statement with an open-ended (or non-existent) WHERE
> clause can be properly isolated (and rolled back properly in the case of a
> failure).
>

PostgreSQL doesn't do gap locks, but instead you have to deal with
http://wiki.postgresql.org/wiki/SSI : the transaction that is deleting 1M
rows, for instance, will have a query that may return rows which another
transaction is adding; if so one of the two will be rolled back. This is in
many ways equivalent from the point of view of writing good SQL that will
work well on both systems.

> In any case, MySQL is certainly a production-capable database like
> PostgreSQL. It has its quirks and downsides, as does any system, including
> PostgreSQL. Biases and false assumptions should be set aside. ;)
>
> Best,
> -jay
>
>
> __**_
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.**org 
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev
>

-- 
Robert Collins 
Distinguished Technologist
HP Cloud Services
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Campus Network Blueprint

2013-07-09 Thread Filipe Manco

Hi Kyle

I already looked at the blueprints and I want to integrate my work with
ML2, mainly because I want users to able to keep the traditional networking
models working in parallel (and I think ML2 is the best way to do this
comparing with the meta plugin).

To be honest, the integration with Neutron and the ML2 was what took most
of the time when writing the blueprint, but although I talk about it on the
blueprint I still not sure about the best way to do it.

About the concept of segments I don't think they fit. From what I
understand a segment, in the context of ML2, represents part of the virtual
network that uses some technology to connect a set of VMs, so if I delete a
segment those VMs will loose connectivity. In the context of my blueprint a
segment (or cSegment as I called it) represents a domain where I can create
virtual links across a set of nodes. If I delete a cSegment that doesn't
mean any VM will loose connectivity, what will happen is that the network
is remapped and the traffic will cross another cSegment or another set of
cSegments.

Something that I would like you (or other guys from ML2) to comment is how
to integrate new operations (beyond the ones related with the base data
model) in the ML2 interfaces. The MechanismDriver interface supports
operations on ports and networks, but as you can see I have new entities.
My idea is that there should be no changes to the original interfaces,
because this are specific to a type of network. What do you think?

It would help me if you have the time to take a look at the blueprint and
comment on the ML2 parts.

Thanks for your help!

Filipe Manco
http://about.me/fmanco

2013/7/9 Kyle Mestery (kmestery) 

> On Jul 9, 2013, at 11:18 AM, Filipe Manco  wrote:
> >
> > Hi all,
> >
> > Just published a blueprint for some work I'm starting right now. I would
> appreciate if someone can take a look and give some comments.
> >
> > https://blueprints.launchpad.net/neutron/+spec/campus-network
> >
> > Thank you
> >
> Hi Filipe:
>
> At first glance, there appears to be a lot of overlap with ML2 here. Have
> you looked at the ML2 blueprints, and the specific use case of
> multi-segment networks? The ML2 team is working hard to close ML2
> blueprints in H2/H3, perhaps some of your ideas could be incorporated here?
>
> Here is the ML2 Meeting page which has links to the blueprints we're
> tracking:
>
> https://wiki.openstack.org/wiki/Meetings/ML2
>
> Thanks,
> Kyle
>
> >
> > Filipe Manco
> > http://about.me/fmanco
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Proposal for new Program: OpenStack Deployment

2013-07-09 Thread Robert Collins

Official Title: OpenStack Deployment
PTL: Robert Collins 
Distinguished Technologist
HP Cloud Services
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] awareness of nova cells, availability zones etc.

2013-07-09 Thread Russell Bryant

On 07/09/2013 07:42 PM, Angus Salkeld wrote:
> On 09/07/13 11:59 -0400, Eoghan Glynn wrote:
>>
>> Hi Folks,
>>
>> Just wondering to what extent Heat deployment policies can be made
>> nova cell/AZ-aware?
>>
>> For example, one could envisage a situation in which it would sense
>> for an autoscaling group to have cell-affinity, but also have AZ-
>> anti-affinity within the cell for resiliance against datacenter
>> outages.
>>
>> Are those kinds of policies currently supported by Heat?
> 
> No.

It seems like the planned instance grouping additions to nova in Havana
would be helpful here.

https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension

This will provide the ability to express to nova that multiple instances
are in a group, and that a policy should be applied to this group, such
as various types of affinity or anti-affinity.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Embrane's Neutron Plugin

2013-07-09 Thread Ivar Lazzaro

Hi,

My name is Ivar Lazzaro, I’m an Italian developer currently employed at Embrane.


Embrane provides L3 to L7 network services, (including routing, load balancing, 
SSL offloads, firewalls and IPsec VPNs), and we have developed a Neutron plugin 
that we would like to share and contribute to Openstack[1].


My experience with OpenStack started with the Essex edition, which I deployed 
and managed as a "user". Embrane leverages any existing form of L2 to offer 
connectivity at L3 and above, and therefore our interest in contributing to 
OpenStack grew as L3 (and above) capabilities started to be added to Neutron, 
leading to the realization of a Neutron plugin.


I'd like to talk about it with you before "blindly" requesting a review, and 
get your feedback and advice in order to improve it at the most!


The idea is to provide L3 connectivity in Openstack through our software 
platform, called heleos, obviously using a plugin to follow the Neutron 
workflow.Since we don't provide L2 connectivity (which is part of the core APIs 
as well) our plugin is going to work together with one of the existing, which 
will manage L2 connectivity and share all the information needed.


Therefore, whenever a user chooses to use Embrane's Neutron plugin, he 
specifies one of the supported existing plugins in the configuration file, and 
L2 connectivity will be provided by that specific choice.

At the current state, for instance, our plugin is able to work with the 
OpenVSwitch's so that:


-create_network() will call OVS plugin;

-create_port() will call OVS plugin;

-crate_router() will call Embrane's which will use knowledge from the OVS 
plugin in order to provide L3 connectivity.


and so forth...

The calls can be asynchronous (using Router "status" in a way similar to the 
LBaaS extension).


Without going too much into details, that's all about the L3 plugin that we 
would like to share. We are also interested in sharing a LBaaS service plugin, 
but I'll do a different blueprint for that one.

All your feedback and comments are welcome :)


Thanks,

Ivar.


[1] https://blueprints.launchpad.net/neutron/+spec/embrane-neutron-plugin
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Service Type Framework implementation

2013-07-09 Thread Mark McClain

On Jul 9, 2013, at 5:37 PM, Nachi Ueno  wrote:
> 
> We have two suboption for db api based solution
> 
> Option4. REST API + DB with Preload with Conf
> https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf14b7b30_00
> 
> so IMO, we can drop  option3.
> 
> I believe option4 is easy to implement.
> 

I'm not onboard with option 4 either.  At the last summit, we talked about 
making Neutron easier to deploy.  Using a database to sync configuration state 
adds complexity.  Having some values in a configuration and others in the 
database (even cached) is a recipe for a major headache.  For the deployments 
running multiple instances of Neutron, they should be using Chef, Chef, Salt, 
etc for managing their configs anyway.

Using only configuration files (option 1) remains my preference.

mark 
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [jenkins] [tempest] Trouble with a patch

2013-07-09 Thread Jeremy Stanley

On 2013-07-09 23:55:03 + (+), Kyle Mestery (kmestery) wrote:
> Thanks for the note Jeremy! Can you shoot another reply on this
> thread so those of us with patches failing can resubmit?

Yep, now that https://review.openstack.org/36349 has merged you
should be safe to recheck any proposed changes or reverify any
approved changes which failed with that error.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [jenkins] [tempest] Trouble with a patch

On Jul 9, 2013, at 6:34 PM, Jeremy Stanley  wrote:
> On 2013-07-09 22:01:15 + (+), Kyle Mestery (kmestery) wrote:
>> I'm currently hitting issues with a patch [1] not passing Jenkins
>> due to a Horizon python-keystonclient dependency issue in no way
>> related to my change.
> [...]
> 
> It should be resolved shortly (the change is in the gate pipeline
> already). Long story short, horizon and one of horizon's
> non-OpenStack dependencies disagreed over which versions of
> python-keystoneclient they needed in such a way that there was no
> resolvable overlap.


Thanks for the note Jeremy! Can you shoot another reply on this
thread so those of us with patches failing can resubmit?

Thanks!
Kyle

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] awareness of nova cells, availability zones etc.

2013-07-09 Thread Angus Salkeld


On 09/07/13 11:59 -0400, Eoghan Glynn wrote:


Hi Folks,

Just wondering to what extent Heat deployment policies can be made
nova cell/AZ-aware?

For example, one could envisage a situation in which it would sense
for an autoscaling group to have cell-affinity, but also have AZ-
anti-affinity within the cell for resiliance against datacenter
outages.

Are those kinds of policies currently supported by Heat?


No.

-Angus



Cheers,
Eoghan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [jenkins] [tempest] Trouble with a patch

2013-07-09 Thread Jeremy Stanley

On 2013-07-09 22:01:15 + (+), Kyle Mestery (kmestery) wrote:
> I'm currently hitting issues with a patch [1] not passing Jenkins
> due to a Horizon python-keystonclient dependency issue in no way
> related to my change.
[...]

It should be resolved shortly (the change is in the gate pipeline
already). Long story short, horizon and one of horizon's
non-OpenStack dependencies disagreed over which versions of
python-keystoneclient they needed in such a way that there was no
resolvable overlap.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

2013-07-09 Thread Jay Pipes


On 07/08/2013 05:18 PM, Sean Dague wrote:

On 07/01/2013 01:35 PM, Clint Byrum wrote:

The way the new keystone-manage command "token_flush" works right now
is quite broken by MySQL and InnoDB's gap locking behavior:

https://bugs.launchpad.net/1188378

Presumably other SQL databases like PostgreSQL will have similar problems
with doing massive deletes, but I am less familiar with them.

I am trying to solve this in keystone, and my first attempt is here:

https://review.openstack.org/#/c/32044/

However, MySQL does not support using "LIMIT" in a sub-query that
is feeding an IN() clause, so that approach will not work. Likewise,
sqlalchemy does not support the MySQL specific extension to DELETE which
allows it to have a LIMIT clause.

Now, I can do some hacky things, like just deleting all of the expired
tokens from the oldest single second, but that could also potentially
be millions of tokens, and thus, millions of gaps to lock.

So, there is just not one way to work for all databases, and we have to
have a special mode for MySQL.

I was wondering if anybody has suggestions and/or examples of how to do
that with sqlalchemy.


Honestly, my answer is typically to ask Jay, he understands a lot of the
ways to get SQLA to do the right thing in mysql.


LOL, /me blushes.

In this case, I'd propose something like this, which should work fine 
for any database:


cutoff = timeutils.utcnow() - 60  # one minute ago...

# DELETE in 500 record chunks
q = session.query(
TokenModel.id).filter(
TokenModel.expires < cutoff)).limit(500)
while True:
results = q.all()
if len(results):
ids_to_delete = [r[0] for r in results]
session.query(TokenModel).filter(
TokenModel.id.in_(ids_to_delete)).delete()
else:
break

Code not tested, use with caution, YMMV, etc etc...

Best,
-jay




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program Proposal: refstack

2013-07-09 Thread Tom Fifield


On 10/07/13 01:01, Monty Taylor wrote:

Hey all,

I'd like to propose an official program to the TC - refstack, a program
for verifying interoperability between implementations via FITS testing.

Official Title: OpenStack Interoperability
Initial PTL: Monty Taylor 
Mission Statement:
   Develop and maintain FITS testing and interoperability reporting for
   OpenStack deployments and distributions.


No problems with the program, but I would caution the title. 
'Interoperability' is much broader than 'interoperability between 
OpenStack clouds'. This could be a potential point of confusion for 
those who are looking for other kinds of interoperability - unless you 
want to take those on too ? ;)



Regards,


Tom

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

2013-07-09 Thread Jay Pipes


On 07/08/2013 08:32 PM, Adam Young wrote:

On 07/08/2013 04:35 PM, Clint Byrum wrote:

Excerpts from Adam Young's message of 2013-07-08 13:18:55 -0700:

On 07/01/2013 01:35 PM, Clint Byrum wrote:

The way the new keystone-manage command "token_flush" works right now
is quite broken by MySQL and InnoDB's gap locking behavior:

https://bugs.launchpad.net/1188378

Presumably other SQL databases like PostgreSQL will have similar
problems
with doing massive deletes, but I am less familiar with them.

I am trying to solve this in keystone, and my first attempt is here:

https://review.openstack.org/#/c/32044/

However, MySQL does not support using "LIMIT" in a sub-query that
is feeding an IN() clause, so that approach will not work. Likewise,
sqlalchemy does not support the MySQL specific extension to DELETE
which
allows it to have a LIMIT clause.

Now, I can do some hacky things, like just deleting all of the expired
tokens from the oldest single second, but that could also potentially
be millions of tokens, and thus, millions of gaps to lock.

So, there is just not one way to work for all databases, and we have to
have a special mode for MySQL.

I was wondering if anybody has suggestions and/or examples of how to do
that with sqlalchemy.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

In general, if you have millions of roles, you need a real database.  I
would not try to work through SQL Alchemy for this. Instead, you
probably just want to make sure that the token_flush is run fairly
regularly on your database.


I'm not sure I understand you.

* I am asking about millions of tokens, not roles.

Heh, I mean Rows, and somehow type roles.


* I am asking about MySQL.. presumably a "real" database.

  I have to admit I am a bit of a Postgresql Bigot. I don't really
consider MySQL a real database, althought it has improved a lot over the
years.


As has PostgreSQL. VACUUM, anyone?

>   I am not up to speed on"InnoDB's gap locking behavior" but it is

not something I would expect to be a problem in Postgresql.


InnoDB and PostgreSQL behave in very different manners regarding locking 
and transaction isolation, even though they both implement a version of 
MVCC. In InnoDB's case, its implementation of MVCC is optimized more for 
storage space (it allows using a series of log records to reconstruct or 
undo a record to a particular "version" of the record) vs. PostgreSQL, 
which stores every version of every record in its data space.


AFAIK, PostgreSQL won't issue gap locks like InnoDB because it will 
simply write a new version of the records -- a version that marks the 
record as deleted -- to its data files, instead of gap locking to ensure 
that rows affected by a DELETE statement with an open-ended (or 
non-existent) WHERE clause can be properly isolated (and rolled back 
properly in the case of a failure).


In any case, MySQL is certainly a production-capable database like 
PostgreSQL. It has its quirks and downsides, as does any system, 
including PostgreSQL. Biases and false assumptions should be set aside. ;)


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [Horizon] Heat Resource Topology Demo

2013-07-09 Thread Tim Schnell

Hey Steve,

Thanks, I have been trying to figure out the best place to put it and I agree, 
I'll add it to the Details tab. For now I just pushed it up as a tab in the 
dashboard but I'll move it.
https://review.openstack.org/#/c/36351/

[cid:BF160E9B-446E-48E5-AF4E-1A91CE1F2B75]

From: Steve Baker mailto:sba...@redhat.com>>
Reply-To: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>
Date: Tuesday, July 9, 2013 4:39 PM
To: 
"openstack-dev@lists.openstack.org" 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Heat] [Horizon] Heat Resource Topology Demo

On 07/10/2013 04:59 AM, Tim Schnell wrote:
Just a quick demo of the Resource Topology. I'm working on getting it into Code 
Review for Horizon either today or tomorrow.
http://www.youtube.com/watch?v=pe9EM8WhLPA


Very nice, looking forward to this in Horizon.

I've seen Tim mention that this would go into its own tab on the stack page.  
Personally I'd like to see it at the top of the stack Details tab, making the 
Details tab more of a dashboard for the running stack.

cheers
<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [jenkins] [tempest] Trouble with a patch

Also the ML2 Mechansim Driver review from Andre:

https://review.openstack.org/#/c/33201/
http://logs.openstack.org/33201/15/check/gate-tempest-devstack-vm-neutron/1140/console.html

Thanks,
Kyle

On Jul 9, 2013, at 5:12 PM, Nachi Ueno  wrote:

> Hi Kyle
> 
> I'm facing same trouble also.
> 
> [1] https://review.openstack.org/#/c/33148/
> [2] 
> http://logs.openstack.org/33148/19/check/gate-tempest-devstack-vm-neutron/1144/console.html
> 
> Thanks
> Nachi
> 
> 2013/7/9 Kyle Mestery (kmestery) :
>> I'm currently hitting issues with a patch [1] not passing Jenkins due to a 
>> Horizon python-keystonclient dependency issue in no way related to my 
>> change. I've rerun the patch twice, and it still fails the same way. See the 
>> bottom of the console output here [2]. I need some guidance here, as this 
>> particular error doesn't appear to be related at all to my change and is 
>> blocking my patch.
>> 
>> Thanks,
>> Kyle
>> 
>> [1] https://review.openstack.org/#/c/35384/
>> [2] 
>> http://logs.openstack.org/35384/8/check/gate-tempest-devstack-vm-neutron/1136/console.html
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [jenkins] [tempest] Trouble with a patch

Hi Kyle

I'm facing same trouble also.

[1] https://review.openstack.org/#/c/33148/
[2] 
http://logs.openstack.org/33148/19/check/gate-tempest-devstack-vm-neutron/1144/console.html

Thanks
Nachi

2013/7/9 Kyle Mestery (kmestery) :
> I'm currently hitting issues with a patch [1] not passing Jenkins due to a 
> Horizon python-keystonclient dependency issue in no way related to my change. 
> I've rerun the patch twice, and it still fails the same way. See the bottom 
> of the console output here [2]. I need some guidance here, as this particular 
> error doesn't appear to be related at all to my change and is blocking my 
> patch.
>
> Thanks,
> Kyle
>
> [1] https://review.openstack.org/#/c/35384/
> [2] 
> http://logs.openstack.org/35384/8/check/gate-tempest-devstack-vm-neutron/1136/console.html
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] [jenkins] [tempest] Trouble with a patch

I'm currently hitting issues with a patch [1] not passing Jenkins due to a 
Horizon python-keystonclient dependency issue in no way related to my change. 
I've rerun the patch twice, and it still fails the same way. See the bottom of 
the console output here [2]. I need some guidance here, as this particular 
error doesn't appear to be related at all to my change and is blocking my patch.

Thanks,
Kyle

[1] https://review.openstack.org/#/c/35384/
[2] 
http://logs.openstack.org/35384/8/check/gate-tempest-devstack-vm-neutron/1136/console.html
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] common requirements

2013-07-09 Thread John Griffith

Hey All,

I ran into an issue with Cinder the other week and it turned out it was due
to some conflicts with keystoneclient version.

Long story short, the issue is that Cinder did the straight sync over from
the common-requirements file which placed an upper bound on
keystoneclient.  It looks like other projects haven't done this and most
don't have the upper bound set.  There are other mismatches however I was
only working on the one.

I was going to log a bug against all the projects to do a sync of
requirements and test-requirments to get us all aligned.  Before doing so I
thought I should check to see if there are known cases where the settings
in the common requirements file don't work for you currently?  I understand
that horizon for example has some upcoming requirements for keystonclient
0.3.0 so that could be a reason to update the req's file first.

Anyway if there are issues that folks already know about then I suppose the
answer is to get that fixed up first and then we can start working towards
getting all of the projects in sync.

Thoughts, agreement, disagreement etc?

Thanks,
John
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [Horizon] Heat Resource Topology Demo

2013-07-09 Thread Steve Baker

On 07/10/2013 04:59 AM, Tim Schnell wrote:
> Just a quick demo of the Resource Topology. I'm working on getting it
> into Code Review for Horizon either today or tomorrow.
> http://www.youtube.com/watch?v=pe9EM8WhLPA
>
>
Very nice, looking forward to this in Horizon.

I've seen Tim mention that this would go into its own tab on the stack
page.  Personally I'd like to see it at the top of the stack Details
tab, making the Details tab more of a dashboard for the running stack.

cheers
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Service Type Framework implementation

Hi folks

Replied Inline

2013/7/9 Salvatore Orlando :
> Some comments inline.
>
> Salvatore
>
>
> On 9 July 2013 21:58, Eugene Nikanorov  wrote:
>>
>> Nachi,
>>
>> I think that dynamic loading/preloaded modules/REST api analogs of nova
>> flavor is a bit too forward looking in comparison to what I'm trying to
>> solve right now with existing patch.
>
>
> Besides, the real issue with this approach is that neutron would be lending
> itself to any sort of security exploit. I am not a security expert, so feel
> free to disagree if you want.
> I would just prefer to not see dynamic loading of python modules of values
> which are stored in the db; not in this release, not in the next one.

We have two suboption for db api based solution

Option3. DB API + Dynamic module load (many guys are saying no for this option3)
https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf0f4e2a2_1163

Option4. REST API + DB with Preload with Conf
https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf14b7b30_00

so IMO, we can drop  option3.

I believe option4 is easy to implement.

>>
>>
>> I think what really matters is how service providers are referenced from
>> other resources.
>>
>> 1) From logic perspective service provider could be referenced by
>> (service_type, name) as it's unique primary key.
>> 2) From data normalization perspective it's better (and more convenient)
>> to have an unique ID in resource provider model.
>
>
> Adding another primary key were you already identified a couple of
> attributes which are a primary key is actually, from what I recall from my
> studies, de-normalization.
> Sorry, this was just pedant me talking. Feel free to ignore.
>
>>
>> Obviously having ID works for DB implementation and doesn't work for
>> in-memory implementation.
>> In other words, we can't use ID if we go with in-memory implementation.
>
>
> You could, but it would not make a lot of sense; and you would have to store
> those ids somewhere anyway; so - no it's not a good idea.
>
> When you associate an instance of a service to a provider, you might think
> that the fact that they key is (type, name) will force you to use two
> attributes. This would be true if you think about the corresponding E-R
> model. However, in the case of the APIs we're dealing with, the resource
> type itself identifies the first bit of the (type, name) pair. So one might
> as well associate only the service provider name to the service instance.
>
>>
>> 3) From data modelling perspective it's better to have ID in service
>> provider model as referencing models will be simpler and easier to maintain.
>> 4) From CLI perspective it's more convenient if resource has ID, it's a
>> common way of specifying resource.
>
>
> We already clarified that for referencing items in the CLI (or horizon) we
> can use either name or id. It's a consolidated practice in both of them.
>
>>
>> 5) From user perspective it's more convenient to specify the name of
>> service provider.
>> But that is usually solved either by Horizon or by cli, like it's done for
>> networks/subnets where name of the object is specified.
>>
>> Resuming all this I see significant benefits of using ID (and hence, db
>> implementation) over not using it.
>> Also, I don't think storing immutable data in db is any kind of a bad
>> design: it's just a storage anyway.
>
>
> As Mark has rightly pointed out, it's generally not a great idea to store
> configuration data in the db.
> However in this case it is worth mentioning that the data in the db is
> exactly the same as the data in the config files.

If same data are stored in the multiple place, IMO it is not good idea.
In current implementation, the data is stored in DB + on each neutron
servers conf.
Who's master in this case??
In current implementation, recent load of conf wins, so it could be
disaster for operators.

(I draw figure for this case)
https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf0f4e2a2_1136


>>
>> DB storage offers better integration with other objects stored in db, and
>> saves some code lines doing stuff which DB normally does.
>> That lines will stack up in case we add more objects (like service
>> offerings) on top of in-memory storage.

so how about this option?

Option4. REST API + DB with Preload with Conf
https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf14b7b30_00

Best
Nachi

>
>>
>> Thanks,
>> Eugene.
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Jul 9, 2013 at 11:00 PM, Nachi Ueno  wrote:
>>>
>>> Hi Eugene
>>>
>>> I agree for dynamic loading is difficult to implement.
>>> (mainly for security perspective)
>>>
>>> Salvatore looks clearly for no for dynamic loading.
>>>
>>> So I added another option.
>>> how about have list of preloaded module in the conf?
>>> and setup service type from REST API such as nova flavor api
>>>
>>>
>>> https

Re: [openstack-dev] Program description for Oslo

2013-07-09 Thread Dolph Mathews

On Tue, Jul 9, 2013 at 3:22 PM, Monty Taylor  wrote:

>
>
> On 07/09/2013 11:43 AM, Mark McLoughlin wrote:
> > On Tue, 2013-07-09 at 11:39 -0400, Doug Hellmann wrote:
> >>
> >>
> >>
> >> On Tue, Jul 9, 2013 at 6:11 AM, Mark McLoughlin 
> wrote:
> >> Hey
> >>
> >> The mission statement is what we've been using for a while. The
> >> "official title" is new.
> >>
> >>   Official Title: OpenStack Common Libraries
> >>   PTL: Mark McLoughlin 
> >>   Mission Statement:
> >> To produce a set of python libraries containing
> infrastructure code
> >> shared by OpenStack projects. The APIs provided by these
> libraries
> >> should be high quality, stable, consistent and generally
> applicable.
> >>
> >>
> >> +1
> >>
> >>
> >> What sort of code would not be "infrastructure" but would be "shared
> >> by OpenStack projects"? Is the "infrastructure" redundant?
> >
> > Simply saying "code" isn't enterprisey enough
> >
> > (Point taken, removed :)
>
>
> Can you add synergy?
>

+1 for renaming 'oslo' to 'synergism'


>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo



On 07/09/2013 11:43 AM, Mark McLoughlin wrote:
> On Tue, 2013-07-09 at 11:39 -0400, Doug Hellmann wrote:
>>
>>
>>
>> On Tue, Jul 9, 2013 at 6:11 AM, Mark McLoughlin  wrote:
>> Hey
>> 
>> The mission statement is what we've been using for a while. The
>> "official title" is new.
>> 
>>   Official Title: OpenStack Common Libraries
>>   PTL: Mark McLoughlin 
>>   Mission Statement:
>> To produce a set of python libraries containing infrastructure 
>> code
>> shared by OpenStack projects. The APIs provided by these 
>> libraries
>> should be high quality, stable, consistent and generally 
>> applicable.
>>
>>
>> +1
>>
>>
>> What sort of code would not be "infrastructure" but would be "shared
>> by OpenStack projects"? Is the "infrastructure" redundant?
> 
> Simply saying "code" isn't enterprisey enough
> 
> (Point taken, removed :)


Can you add synergy?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Current biggest OpenStack gate fail culprit - neutron bug #1194026

2013-07-09 Thread Salvatore Orlando

It's been set to 'High' now.
Unfortunately I cannot give this bug the attention it deserves now, so I've
not taken it.

Adding quantum/neutron-core to trigger attention of other core devs.



On 9 July 2013 19:40, Sean Dague  wrote:

> On 07/09/2013 12:20 PM, Thierry Carrez wrote:
>
>> Monty Taylor wrote:
>>
>>> I think having an uber-triager group comprised of the small set of gate
>>> watchers is not a terrible idea. Have the group, have that group be a
>>> member of all the other driver groups.
>>>
>>
>> FWIW you actually just need to be a bug supervisor to set priority,
>> which in most cases means adding yourself to the PROJECT-bugs group on
>> LP. Some of those groups are still moderated, but most of them are open.
>>
>
> neutron-bugs is a moderated group.
>
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __**_
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.**org 
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Service Type Framework implementation

2013-07-09 Thread Salvatore Orlando

Some comments inline.

Salvatore

On 9 July 2013 21:58, Eugene Nikanorov  wrote:

> Nachi,
>
> I think that dynamic loading/preloaded modules/REST api analogs of nova
> flavor is a bit too forward looking in comparison to what I'm trying to
> solve right now with existing patch.
>

Besides, the real issue with this approach is that neutron would be lending
itself to any sort of security exploit. I am not a security expert, so feel
free to disagree if you want.
I would just prefer to not see dynamic loading of python modules of values
which are stored in the db; not in this release, not in the next one.

>
> I think what really matters is how service providers are referenced from
> other resources.
>
> 1) From logic perspective service provider could be referenced by
> (service_type, name) as it's unique primary key.
> 2) From data normalization perspective it's better (and more convenient)
> to have an unique ID in resource provider model.
>

Adding another primary key were you already identified a couple of
attributes which are a primary key is actually, from what I recall from my
studies, de-normalization.
Sorry, this was just pedant me talking. Feel free to ignore.

> Obviously having ID works for DB implementation and doesn't work for
> in-memory implementation.
> In other words, we can't use ID if we go with in-memory implementation.
>

You could, but it would not make a lot of sense; and you would have to
store those ids somewhere anyway; so - no it's not a good idea.

When you associate an instance of a service to a provider, you might think
that the fact that they key is (type, name) will force you to use two
attributes. This would be true if you think about the corresponding E-R
model. However, in the case of the APIs we're dealing with, the resource
type itself identifies the first bit of the (type, name) pair. So one might
as well associate only the service provider name to the service instance.

> 3) From data modelling perspective it's better to have ID in service
> provider model as referencing models will be simpler and easier to maintain.
> 4) From CLI perspective it's more convenient if resource has ID, it's a
> common way of specifying resource.
>

We already clarified that for referencing items in the CLI (or horizon) we
can use either name or id. It's a consolidated practice in both of them.

> 5) From user perspective it's more convenient to specify the name of
> service provider.
> But that is usually solved either by Horizon or by cli, like it's done for
> networks/subnets where name of the object is specified.
>
> Resuming all this I see significant benefits of using ID (and hence, db
> implementation) over not using it.
> Also, I don't think storing immutable data in db is any kind of a bad
> design: it's just a storage anyway.
>

As Mark has rightly pointed out, it's generally not a great idea to store
configuration data in the db.
However in this case it is worth mentioning that the data in the db is
exactly the same as the data in the config files.

> DB storage offers better integration with other objects stored in db, and
> saves some code lines doing stuff which DB normally does.
> That lines will stack up in case we add more objects (like service
> offerings) on top of in-memory storage.
>

> Thanks,
> Eugene.
>
>
>
>
>
>
>
> On Tue, Jul 9, 2013 at 11:00 PM, Nachi Ueno  wrote:
>
>> Hi Eugene
>>
>> I agree for dynamic loading is difficult to implement.
>> (mainly for security perspective)
>>
>> Salvatore looks clearly for no for dynamic loading.
>>
>> So I added another option.
>> how about have list of preloaded module in the conf?
>> and setup service type from REST API such as nova flavor api
>>
>>
>> https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf14b7b30_00
>>
>> NOTE: I updated the style of doc
>>
>> Best
>> Nachi
>>
>>
>> 2013/7/9 Eugene Nikanorov :
>> > Hi Nachi,
>> >
>> > I agree on the future plan.
>> > However, dynamic loading/unloading of provider drivers will require
>> > additional code in service plugins, I'm not sure this will be fully
>> > supported in Havana (while I'm totally agree on implementing it)
>> >
>> > Thanks,
>> > Eugene.
>> >
>> >
>> > On Tue, Jul 9, 2013 at 3:40 AM, Nachi Ueno  wrote:
>> >>
>> >> Hi Eugene
>> >>
>> >> It still not make sense for me to store static configuration on the DB
>> >> just for easy implementation.
>> >> However if the service type will support creation and deletion REST
>> >> api in future, I would like to approve this patch
>> >> as a first step of it.
>> >> You answered "I think it's doable but I'd still consider current
>> >> implementation as a first step - enikanorov. "
>> >> in the googled docs. so I believe we are in the same boat now.
>> >>
>> >> I wanna make it clear future work.
>> >>
>> >> - Service Type REST API (for admin) will add supports
>> >>   - Ceate Service Type
>> >>   - Delete Service Type
>> >>  -  Each driver users will lazy load the libr

Re: [openstack-dev] [Neutron] Service Type Framework implementation

2013-07-09 Thread Eugene Nikanorov

Nachi,

I think that dynamic loading/preloaded modules/REST api analogs of nova
flavor is a bit too forward looking in comparison to what I'm trying to
solve right now with existing patch.

I think what really matters is how service providers are referenced from
other resources.

1) From logic perspective service provider could be referenced by
(service_type, name) as it's unique primary key.
2) From data normalization perspective it's better (and more convenient) to
have an unique ID in resource provider model.
Obviously having ID works for DB implementation and doesn't work for
in-memory implementation.
In other words, we can't use ID if we go with in-memory implementation.
3) From data modelling perspective it's better to have ID in service
provider model as referencing models will be simpler and easier to maintain.
4) From CLI perspective it's more convenient if resource has ID, it's a
common way of specifying resource.
5) From user perspective it's more convenient to specify the name of
service provider.
But that is usually solved either by Horizon or by cli, like it's done for
networks/subnets where name of the object is specified.

Resuming all this I see significant benefits of using ID (and hence, db
implementation) over not using it.
Also, I don't think storing immutable data in db is any kind of a bad
design: it's just a storage anyway.
DB storage offers better integration with other objects stored in db, and
saves some code lines doing stuff which DB normally does.
That lines will stack up in case we add more objects (like service
offerings) on top of in-memory storage.

Thanks,
Eugene.

On Tue, Jul 9, 2013 at 11:00 PM, Nachi Ueno  wrote:

> Hi Eugene
>
> I agree for dynamic loading is difficult to implement.
> (mainly for security perspective)
>
> Salvatore looks clearly for no for dynamic loading.
>
> So I added another option.
> how about have list of preloaded module in the conf?
> and setup service type from REST API such as nova flavor api
>
>
> https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf14b7b30_00
>
> NOTE: I updated the style of doc
>
> Best
> Nachi
>
>
> 2013/7/9 Eugene Nikanorov :
> > Hi Nachi,
> >
> > I agree on the future plan.
> > However, dynamic loading/unloading of provider drivers will require
> > additional code in service plugins, I'm not sure this will be fully
> > supported in Havana (while I'm totally agree on implementing it)
> >
> > Thanks,
> > Eugene.
> >
> >
> > On Tue, Jul 9, 2013 at 3:40 AM, Nachi Ueno  wrote:
> >>
> >> Hi Eugene
> >>
> >> It still not make sense for me to store static configuration on the DB
> >> just for easy implementation.
> >> However if the service type will support creation and deletion REST
> >> api in future, I would like to approve this patch
> >> as a first step of it.
> >> You answered "I think it's doable but I'd still consider current
> >> implementation as a first step - enikanorov. "
> >> in the googled docs. so I believe we are in the same boat now.
> >>
> >> I wanna make it clear future work.
> >>
> >> - Service Type REST API (for admin) will add supports
> >>   - Ceate Service Type
> >>   - Delete Service Type
> >>  -  Each driver users will lazy load the library if it is not loaded.
> >> (may be this should be implemented on service side such as FW,
> >> LBaaS,VPN)
> >>
> >> - Remove service type configuration from conf
> >>
> >> Is this OK for you guys?
> >>
> >> Thanks
> >> Nachi
> >>
> >>
> >> 2013/7/8 Eugene Nikanorov :
> >> > Hi neutron folks,
> >> >
> >> > There has been a discussion around this patch
> >> > https://review.openstack.org/#/c/29750/ that introduces configuration
> >> > options and db table for storing service providers.
> >> >
> >> > The discussion is about whether we should store configuration in the
> db
> >> > or
> >> > not.
> >> > The brief of discussion has been saved here:
> >> >
> >> >
> https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gefc32ecf_00
> >> > Please share your thoughts on this.
> >> >
> >> > While we may continue to discuss the best approach to this, I'd like
> to
> >> > see
> >> > the patch to be committed first (it seems to be ready) as there are
> >> > other
> >> > features depending on it (NSX distributed router, lbaas, fwaas and
> >> > vpnaas
> >> > possibly).
> >> >
> >> >
> >> > Thanks,
> >> > Eugene.
> >> >
> >> > ___
> >> > OpenStack-dev mailing list
> >> > OpenStack-dev@lists.openstack.org
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >> ___
> >> OpenStack-dev mailing list
> >> OpenStack-dev@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.or

Re: [openstack-dev] heads up - upstream pip/virtualenv/setuptools upgrades may cause some hiccups

2013-07-09 Thread Jay Pipes


On 07/05/2013 09:05 PM, Monty Taylor wrote:

Hey all!

A few things are happening in the world of python that are have an
impact on us. The end story is good, and things are getting both simpler
and better (and in a few weeks - MUCH faster for many of us) ... in the
mean time, there may be some evil bunny rabbits.




Thanks for the detailed information, Monty. Can you further elaborate on 
your prediction above about things becoming MUCH faster for many of us? 
Who is the us?


Thanks!
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Infra] Meeting Tuesday July 9th at 19:00 UTC

2013-07-09 Thread Elizabeth Krumbach Joseph

On Mon, Jul 8, 2013 at 9:25 AM, Elizabeth Krumbach Joseph
 wrote:
> The OpenStack Infrastructure (Infra) team is hosting our weekly
> meeting tomorrow, Tuesday July 9th, at 19:00 UTC in #openstack-meeting

Minutes and logs now available:

Minutes: 
http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-07-09-19.03.html
Minutes (text):
http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-07-09-19.03.txt
Log: 
http://eavesdrop.openstack.org/meetings/infra/2013/infra.2013-07-09-19.03.log.html

-- 
Elizabeth Krumbach Joseph || Lyz || pleia2
http://www.princessleia.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Openstack/Devstack + nginx + scalability

2013-07-09 Thread Dean Troyer

On Tue, Jul 9, 2013 at 1:46 PM, Geronimo Orozco  wrote:
> My initial concern or request was to start  with a minimal setting using
> devstacks with nginx and load balancer if that's is not the right way to
> start then I'll be very thankful on how I can start poking around that
> feature ..

You certainly could use DevStack as the backend for your initial
config/testing work and semi-automate the setup using a script in
devstack/extras.d that will run from stack.sh.  But I think you'll
find it inadequate to support any scale/HA testing and will need to
eventually use something else anyway.

BTW, I will resist strongly any suggestions to add nginx as an option
to DevStack.  We've already got too many choices in there...

dt

--

Dean Troyer
dtro...@gmail.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Service Type Framework implementation

Hi Eugene

I agree for dynamic loading is difficult to implement.
(mainly for security perspective)

Salvatore looks clearly for no for dynamic loading.

So I added another option.
how about have list of preloaded module in the conf?
and setup service type from REST API such as nova flavor api

https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gf14b7b30_00

NOTE: I updated the style of doc

Best
Nachi


2013/7/9 Eugene Nikanorov :
> Hi Nachi,
>
> I agree on the future plan.
> However, dynamic loading/unloading of provider drivers will require
> additional code in service plugins, I'm not sure this will be fully
> supported in Havana (while I'm totally agree on implementing it)
>
> Thanks,
> Eugene.
>
>
> On Tue, Jul 9, 2013 at 3:40 AM, Nachi Ueno  wrote:
>>
>> Hi Eugene
>>
>> It still not make sense for me to store static configuration on the DB
>> just for easy implementation.
>> However if the service type will support creation and deletion REST
>> api in future, I would like to approve this patch
>> as a first step of it.
>> You answered "I think it's doable but I'd still consider current
>> implementation as a first step - enikanorov. "
>> in the googled docs. so I believe we are in the same boat now.
>>
>> I wanna make it clear future work.
>>
>> - Service Type REST API (for admin) will add supports
>>   - Ceate Service Type
>>   - Delete Service Type
>>  -  Each driver users will lazy load the library if it is not loaded.
>> (may be this should be implemented on service side such as FW,
>> LBaaS,VPN)
>>
>> - Remove service type configuration from conf
>>
>> Is this OK for you guys?
>>
>> Thanks
>> Nachi
>>
>>
>> 2013/7/8 Eugene Nikanorov :
>> > Hi neutron folks,
>> >
>> > There has been a discussion around this patch
>> > https://review.openstack.org/#/c/29750/ that introduces configuration
>> > options and db table for storing service providers.
>> >
>> > The discussion is about whether we should store configuration in the db
>> > or
>> > not.
>> > The brief of discussion has been saved here:
>> >
>> > https://docs.google.com/presentation/d/1v0nLTEsFOwWeYpYjpw4qe3QHB5lLZEE_b0TmmR5b7ic/edit#slide=id.gefc32ecf_00
>> > Please share your thoughts on this.
>> >
>> > While we may continue to discuss the best approach to this, I'd like to
>> > see
>> > the patch to be committed first (it seems to be ready) as there are
>> > other
>> > features depending on it (NSX distributed router, lbaas, fwaas and
>> > vpnaas
>> > possibly).
>> >
>> >
>> > Thanks,
>> > Eugene.
>> >
>> > ___
>> > OpenStack-dev mailing list
>> > OpenStack-dev@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Openstack/Devstack + nginx + scalability

2013-07-09 Thread Geronimo Orozco

Hi Monty,


I'll jump over  tripleO team to see what they are doing...


My initial concern or request was to start  with a minimal setting using
devstacks with nginx and load balancer if that's is not the right way to
start then I'll be very thankful on how I can start poking around that
feature ..

Thanks for your reply

On Tue, Jul 9, 2013 at 1:28 PM, Monty Taylor  wrote:

> On 07/09/2013 02:05 PM, Geronimo Orozco wrote:
> > Hi, I am very interested in scalabity and HA with horizon is there any
> > effort to provide nginx to devstack(for start) ?
>
> I'm not sure that devstack is going to be the best place to poke at
> scaling concerns or HA concerns for horizon. (the best way to do  both
> with horizon is to scale out behind a load balancer - there is no local
> state)
>
> It's possible I'm missing something though - is there another reason
> that having nginx support in devstack might be useful?
>
> > Where can I join to that effort ?
>
> The TripleO team is certainly interested in HA and scaleout stories for
> openstack - perhaps drop by #tripleo?
>
> > I just joined to this
> > blueprint:
> https://blueprints.launchpad.net/ubuntu/+spec/servercloud-r-webscale
>
> That blueprint is more about packaging requirements for ubuntu itself.
>
> > Any other interested ?
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Openstack/Devstack + nginx + scalability

On 07/09/2013 02:05 PM, Geronimo Orozco wrote:
> Hi, I am very interested in scalabity and HA with horizon is there any
> effort to provide nginx to devstack(for start) ?

I'm not sure that devstack is going to be the best place to poke at
scaling concerns or HA concerns for horizon. (the best way to do  both
with horizon is to scale out behind a load balancer - there is no local
state)

It's possible I'm missing something though - is there another reason
that having nginx support in devstack might be useful?

> Where can I join to that effort ? 

The TripleO team is certainly interested in HA and scaleout stories for
openstack - perhaps drop by #tripleo?

> I just joined to this
> blueprint: 
> https://blueprints.launchpad.net/ubuntu/+spec/servercloud-r-webscale

That blueprint is more about packaging requirements for ubuntu itself.

> Any other interested ?
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] [Horizon] Heat Resource Topology Demo

2013-07-09 Thread Gabriel Hurley

Very nice, can't wait to get the code review for that. :)

-  Gabriel

From: Tim Schnell [mailto:tim.schn...@rackspace.com]
Sent: Tuesday, July 09, 2013 9:59 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Heat] [Horizon] Heat Resource Topology Demo

Hey everyone,

Just a quick demo of the Resource Topology. I'm working on getting it into Code 
Review for Horizon either today or tomorrow.
http://www.youtube.com/watch?v=pe9EM8WhLPA

Thanks!
[cid:image001.png@01CE7C94.5D6CD2A0]
<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Draft Mission statement for tripleo

2013-07-09 Thread Clint Byrum

Excerpts from Robert Collins's message of 2013-07-09 01:21:34 -0700:
> How about
> >   Develop and maintain tooling and infrastructure able to
> >   deploy OpenStack in production, using OpenStack itself wherever
> >   possible.
> 

+1

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Openstack/Devstack + nginx + scalability

2013-07-09 Thread Geronimo Orozco

Hi, I am very interested in scalabity and HA with horizon is there any
effort to provide nginx to devstack(for start) ?

Where can I join to that effort ?


I just joined to this blueprint:
https://blueprints.launchpad.net/ubuntu/+spec/servercloud-r-webscale


Any other interested ?
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

2013-07-09 Thread Adam Young


On 07/09/2013 11:21 AM, Clint Byrum wrote:

Yes please! Is this something that is in development, or at least recorded
in a bug that someone like me can grab? I would love for that to be the
default behavior.


https://blueprints.launchpad.net/keystone/+spec/no-tokens-in-db

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Current biggest OpenStack gate fail culprit - neutron bug #1194026

2013-07-09 Thread Sean Dague


On 07/09/2013 12:20 PM, Thierry Carrez wrote:

Monty Taylor wrote:

I think having an uber-triager group comprised of the small set of gate
watchers is not a terrible idea. Have the group, have that group be a
member of all the other driver groups.


FWIW you actually just need to be a bug supervisor to set priority,
which in most cases means adding yourself to the PROJECT-bugs group on
LP. Some of those groups are still moderated, but most of them are open.


neutron-bugs is a moderated group.

-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] How to write unit tests for db methods?

2013-07-09 Thread Dolph Mathews

I'm assuming you're referring to testing backend drivers as opposed to
database migrations (tests/test_sql_upgrade.py).

Backend agnostic tests land in tests/test_backend.py. Backend-specific
tests, overrides, etc belong in tests/test_backend_sql.py,
tests/test_backend_kvs.py, etc.

Generally, you can't assume that keystone is backed by a database, however,
as it's entirely possible to deploy without one.

On Tue, Jul 9, 2013 at 10:55 AM, Akshat Kakkar wrote:

> How to write unit tests in keystone for the methods which are directly
> calling the backend db? I understand that for testing purpose it should be
> a *fake db*, but how to do that in keystone?
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>

-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] sqlalchemy 0.8 and Grizzly: heat and cinder failing

2013-07-09 Thread David Ripton

On 07/09/2013 12:46 PM, Thomas Goirand wrote:

On 07/08/2013 08:32 PM, Sean Dague wrote:

On 07/08/2013 04:50 AM, Mark McLoughlin wrote:

On Mon, 2013-07-08 at 15:53 +0800, Thomas Goirand wrote:

Hi,

Since python-sqlalchemy 0.8.2 has been uploaded to Sid, Quantum is
uninstallable there right now (see #715294).

I am wondering: what's wrong with sqlalchemy >= 0.8, so that it is
written explicitly in the requirements that we shouldn't use it? Is
there a chance that having such a version of sqlalchemy will make all of
OpenStack grizzly fail? What are the consequences? Would it be safe to
simply patch the requirements file and ignore this?

Don't really have a comment on the specifics, but ...

The history of the 0.8 cap was the fact that 0.8beta1 (or something
equiv) was uploaded near a freeze point. <0.8 didn't stop 0.8beta1 from
being used (go version numbers).

Also in 0.8 a piece was spun out as a separate library (I don't remember
exactly which), which causes some build fails. Because it was around a
freeze the cap was the right approach.

However, projects really should be getting themselves on 0.8 in the
Havana time frame. AFAIK it's really minor changes to work, so should be
a simple review to bump it up.

-Sean

Indeed, most projects seem to work with the new SQLAlchemy. Though heat
fails with a python crash dump:

File
"/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/sqlalchemy/models.py",
line 32, in
class Json(types.TypeDecorator, types.MutableType):
AttributeError: 'module' object has no attribute 'MutableType'

Indeed, MutableType is gone in SQLAlchemy >= 0.8. I'm therefore unsure
what to do to fix the heat package in Sid... :(
Any help would be appreciated.

Yeah, someone who understands the Heat model code well needs to make
class Json no longer inherit from MutableType. I hope it would be
possible to do that in a backward-compatible way so it kept working with
SQLAlchemy 0.7 in addition to working with 0.8.

There's also a big problem with Cinder:

File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py",
line 91, in runchange
change.run(self.engine, step)
File
"/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line
145, in run
script_func(engine)
File
"/root/src/cinder/build-area/cinder-2013.1.2/cinder/db/sqlalchemy/migrate_repo/versions/002_quota_class.py",
line 42, in upgrade
_warn_on_bytestring=False),
TypeError: __init__() got an unexpected keyword argument 'assert_unicode'

Unit tests aren't run at all, and cinder refuses to install (because I'm
doing the db_sync in the postinst, which fails).

Help there would also be appreciated.

AFAICT, these are the only packages affected by the SQLAlchemy upgrade.

Old openstack DB migrations contained a lot of "convert_unicode=True",
"unicode_error=None" and "_warn_on_bytestring=False" in the Column
creation code. Dan Prince removed these from the Nova migrations in
commit 93dec58156e , when he squashed all the migrations for Grizzly.
Other projects still have them here and there, and that appears to be
what is causing the above error.

I suspect, but haven't proven, that it's possible to get rid of them
(because Nova did) and that getting rid of them will fix this problem
(because Nova doesn't have the problem.) Note that we don't like to
modify database migrations because of compatibility concerns, so a
change like this would need to receive a extra review scrutiny to prove
it couldn't break anything.

I threw up a patch to excise these arguments from Cinder DB migrations.
https://review.openstack.org/36302 I will add some comments about how
scary this type of change is to the review.

--
David Ripton Red Hat drip...@redhat.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Created/Deleted-by Parent-data

2013-07-09 Thread Martins, Tiago

Sorry for this - it went to the wrong list. I'm very embarrassed.

Thanks,
Tiago

From: Martins, Tiago
Sent: terça-feira, 9 de julho de 2013 13:50
To: openstack-dev@lists.openstack.org
Subject: Created/Deleted-by Parent-data

Hi!
I just want to make sure of one thing.

The created/deleted_by has the information of the user performing the action 
and user role, your examples were always a dict like {"user-id": "user-id", 
"role":"role-id"}. But I was looking at the identity api method 
get_roles_for_user_and_domain and it returns a list of roles for a given user 
in a given domain. So our dict looks something like

{"user-id":"uuid-value", "roles-id":["uuid-value","uuid-value"]}

For the parent data, I have to inform the domain admin of a given domain, but a 
domain can have more than one admin and your examples were always a dict like 
{"user-id":"uuid-value", "role":"uuid-value} and a user may have many roles. So 
parent-data may be a dict of dicts:

{"user uuid-value":{ "roles-id":["uuid-value","uuid-value"]}, "user 
uuid-value":{ "roles-id":["uuid-value","uuid-value"]}}

So, how this look for you?

Cheers,
Tiago Martins
Software Engineer
BR R&D

tiago.mart...@hp.com
T +55 51 3205 7673
Hewlett-Packard Company
6681 Ipiranga Ave. Bld 95 - 6th floor
Porto Alegre, RS, 90619-900
Brazil

[HP_Blue_RGB_150_MN]

Please print thoughtfully

<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift debugging / performance - large latencies seen.

That helped -- using tempauth instead of keystone improved performance 
significantly on the openStack cluster.
However performance is still slower compared to my Swift only setup where 
commands are sent directly to the swift node instead of going thru the 
controller node in the openstack  cluster.
How does controller and swift node communications affect swift performance?
I've also noticed that all objects are stored on the 2nd swift node and not the 
1st on the openstack cluster. I'm wondering if that could also be a factor in 
slow performance.

Keystone:
TOTAL
   Count:50  Average requests per second:   9.2
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.067 -   2.5130.390  (  0.604)1.948  (all 
obj sizes)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.067 -   3.3370.430  (  0.695)1.997  (all 
obj sizes)  txdcedb82227654b338daa85751f6d1232
   First-byte latency:  0.070 -   2.5130.542  (  0.749)2.255  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.070 -   2.5140.468  (  0.659)1.997  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   First-byte latency:  0.067 -   1.8840.251  (  0.382)0.695  (   
small objs)  tx2ceec827f3304530b01a0d5993eea2e8
   Last-byte  latency:  0.067 -   3.3370.385  (  0.732)1.884  (   
small objs)  txdcedb82227654b338daa85751f6d1232

Tempauth:
   Count:50  Average requests per second:  65.7
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.006 -   0.0730.014  (  0.015)0.055  (all 
obj sizes)  tx69bf033a246645808b2c6a280e334f15
   Last-byte  latency:  0.006 -   0.2480.047  (  0.070)0.198  (all 
obj sizes)  txb8cf5dc0ce264eb08a1e05edbbf5a40f
   First-byte latency:  0.006 -   0.0730.017  (  0.020)0.072  (
tiny objs)  tx69bf033a246645808b2c6a280e334f15
   Last-byte  latency:  0.006 -   0.2480.053  (  0.072)0.195  (
tiny objs)  txb8cf5dc0ce264eb08a1e05edbbf5a40f
   First-byte latency:  0.006 -   0.0260.010  (  0.005)0.026  (   
small objs)  tx65d1fd4b6ae049bb902442ac4c28ffe9
   Last-byte  latency:  0.006 -   0.2180.040  (  0.066)0.198  (   
small objs)  txbfd6ebc74ed04068affd17c123572a44

Swift Only:
TOTAL
   Count:50  Average requests per second: 397.0
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.003 -   0.0070.005  (  0.001)0.006  (all 
obj sizes)  None
   Last-byte  latency:  0.003 -   0.0460.008  (  0.009)0.029  (all 
obj sizes)  None
   First-byte latency:  0.003 -   0.0070.005  (  0.001)0.007  (
tiny objs)  None
   Last-byte  latency:  0.003 -   0.0460.008  (  0.010)0.027  (
tiny objs)  None
   First-byte latency:  0.004 -   0.0060.005  (  0.001)0.006  (   
small objs)  None
   Last-byte  latency:  0.004 -   0.0430.008  (  0.009)0.029  (   
small objs)  None

From: Chmouel Boudjnah [mailto:chmo...@enovance.com]
Sent: Tuesday, July 09, 2013 8:22 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Swift debugging / performance - large latencies 
seen.

On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim 
mailto:tim.sni...@netapp.com>> wrote:
I have 2 openstack clusters running the Folsom release with multiple Swift 
nodes. I also have a small setup that is running only Swift with a single node. 
 I'm noticing very large Swift I/O latencies (seconds long) on the openstack 
clusters - ssbench output snippet is below. Performance is approximately 
identical on the openstack clusters. The Swift only cluster performs much 
better.

Keystone performance can be pretty awful unless you are using something else 
than the default WSGI container configuration (single process eventlet I 
think). I would suggest you try to run it under apache with multiple process.

See the dicussion at last summit about Keystone performance here :

https://etherpad.openstack.org/havana-keystone-performance

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Ceilometer XSD

2013-07-09 Thread Jobin Raju George

The reason I initiated to contribute the XSD was most of the other
components(nova, keystone, glance, etc.) have their XSD's in the
repository, so assuming they were contributed by people like me(and updated
from time to time by the same or other fellows), I would patch this XSD to
the repository.

However, since XSD's need to be updated as soon as the respective API's
are, I am not sure whether they are updated by dedicated developers. Please
let me know if there are such restrictions. Thanks for your time!


On Tue, Jul 9, 2013 at 8:59 PM, Doug Hellmann
wrote:

> The instructions for getting set up to contribute to OpenStack in general
> are at https://wiki.openstack.org/wiki/How_To_Contribute (maybe you've
> already done that, I'm not sure).
>
> As far as this specific change goes, I would have to understand more about
> what it is that you want added to give specific advice. If we don't have
> the tools to make the XSD, I'm not sure how we will ensure it is kept up to
> date with changes to the API.
>
> Doug
>
>
> On Mon, Jul 8, 2013 at 1:33 PM, Jobin Raju George wrote:
>
>> Hey, all!
>>
>> I have prepared the XSD for ceilometer and would like to contribute it to
>> the repositories on github. Can somebody help me out with the process to do
>> this?
>>
>>
>> On Mon, Jul 8, 2013 at 2:19 PM, Jobin Raju George wrote:
>>
>>> Ok, that's fine. I am currently investigating into it. Lets see if I get
>>> any clues. Thanks for you time!
>>>
>>>
>>> On Mon, Jul 8, 2013 at 1:32 PM, Julien Danjou wrote:
>>>
 On Sat, Jul 06 2013, Jobin Raju George wrote:

 > I am trying to use the meters provided by ceilometer to extract usage
 > values from the VM's deployed using openstack.
 >
 > However, in order to programmatically do this I need the XSD files for
 > ceilometer. I tried googling them, posting them on forums and even on
 > launchpad but have not received any response, i would be great if you
 could
 > provide me with the source/link/file of the XSD's.

 The XML API is provided automatically via WSME¹, and I've no idea if it
 provides any XSD automatically generated or something like that.

 ¹  https://pypi.python.org/pypi/WSME

 --
 Julien Danjou
 /* Free Software hacker * freelance consultant
http://julien.danjou.info */

>>>
>>>
>>>
>>> --
>>>
>>>  Thanks and regards,
>>>
>>> Jobin Raju George
>>>
>>> Third Year, Information Technology
>>>
>>> College of Engineering Pune
>>>
>>> Alternate e-mail: georgejr10...@coep.ac.in
>>>
>>>
>>
>>
>> --
>>
>> Thanks and regards,
>>
>> Jobin Raju George
>>
>> Third Year, Information Technology
>>
>> College of Engineering Pune
>>
>> Alternate e-mail: georgejr10...@coep.ac.in
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>


-- 

Thanks and regards,

Jobin Raju George

Third Year, Information Technology

College of Engineering Pune

Alternate e-mail: georgejr10...@coep.ac.in
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Hyper-V Meeting Minutes

2013-07-09 Thread Peter Pouliot

Hi All,

Here's the minutes for today's meeting.

Minutes:
http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-07-09-16.00.html
Minutes (text): 
http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-07-09-16.00.txt
Log:
http://eavesdrop.openstack.org/meetings/hyper_v/2013/hyper_v.2013-07-09-16.00.log.html



Peter J. Pouliot, CISSP
Senior SDET, OpenStack

Microsoft
New England Research & Development Center
One Memorial Drive,Cambridge, MA 02142
ppoul...@microsoft.com | Tel: +1(857) 453 6436

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] [Horizon] Heat Resource Topology Demo

2013-07-09 Thread Tim Schnell

Hey everyone,

Just a quick demo of the Resource Topology. I'm working on getting it into Code 
Review for Horizon either today or tomorrow.
http://www.youtube.com/watch?v=pe9EM8WhLPA

Thanks!
[cid:F8A7CDE7-7AE1-42CB-A834-D228A925FF7A]
<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Created/Deleted-by Parent-data

2013-07-09 Thread Martins, Tiago

Hi!
I just want to make sure of one thing.

The created/deleted_by has the information of the user performing the action 
and user role, your examples were always a dict like {"user-id": "user-id", 
"role":"role-id"}. But I was looking at the identity api method 
get_roles_for_user_and_domain and it returns a list of roles for a given user 
in a given domain. So our dict looks something like

{"user-id":"uuid-value", "roles-id":["uuid-value","uuid-value"]}



For the parent data, I have to inform the domain admin of a given domain, but a 
domain can have more than one admin and your examples were always a dict like 
{"user-id":"uuid-value", "role":"uuid-value} and a user may have many roles. So 
parent-data may be a dict of dicts:

{"user uuid-value":{ "roles-id":["uuid-value","uuid-value"]}, "user 
uuid-value":{ "roles-id":["uuid-value","uuid-value"]}}



So, how this look for you?

Cheers,
Tiago Martins
Software Engineer
BR R&D

tiago.mart...@hp.com
T +55 51 3205 7673
Hewlett-Packard Company
6681 Ipiranga Ave. Bld 95 - 6th floor
Porto Alegre, RS, 90619-900
Brazil

[HP_Blue_RGB_150_MN]

Please print thoughtfully

<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] sqlalchemy 0.8 and Grizzly: heat and cinder failing

2013-07-09 Thread Thomas Goirand

On 07/08/2013 08:32 PM, Sean Dague wrote:
> On 07/08/2013 04:50 AM, Mark McLoughlin wrote:
>> On Mon, 2013-07-08 at 15:53 +0800, Thomas Goirand wrote:
>>> Hi,
>>>
>>> Since python-sqlalchemy 0.8.2 has been uploaded to Sid, Quantum is
>>> uninstallable there right now (see #715294).
>>>
>>> I am wondering: what's wrong with sqlalchemy >= 0.8, so that it is
>>> written explicitly in the requirements that we shouldn't use it? Is
>>> there a chance that having such a version of sqlalchemy will make all of
>>> OpenStack grizzly fail? What are the consequences? Would it be safe to
>>> simply patch the requirements file and ignore this?
>>
>> Don't really have a comment on the specifics, but ...
> 
> The history of the 0.8 cap was the fact that 0.8beta1 (or something
> equiv) was uploaded near a freeze point. <0.8 didn't stop 0.8beta1 from
> being used (go version numbers).
> 
> Also in 0.8 a piece was spun out as a separate library (I don't remember
> exactly which), which causes some build fails. Because it was around a
> freeze the cap was the right approach.
> 
> However, projects really should be getting themselves on 0.8 in the
> Havana time frame. AFAIK it's really minor changes to work, so should be
> a simple review to bump it up.
> 
> -Sean

Indeed, most projects seem to work with the new SQLAlchemy. Though heat
fails with a python crash dump:

  File
"/home/zigo/sources/openstack/grizzly/heat/build-area/heat-2013.1.2/heat/db/sqlalchemy/models.py",
line 32, in 
class Json(types.TypeDecorator, types.MutableType):
AttributeError: 'module' object has no attribute 'MutableType'

Indeed, MutableType is gone in SQLAlchemy >= 0.8. I'm therefore unsure
what to do to fix the heat package in Sid... :(
Any help would be appreciated.

There's also a big problem with Cinder:

  File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py",
line 91, in runchange
change.run(self.engine, step)
  File
"/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line
145, in run
script_func(engine)
  File
"/root/src/cinder/build-area/cinder-2013.1.2/cinder/db/sqlalchemy/migrate_repo/versions/002_quota_class.py",
line 42, in upgrade
_warn_on_bytestring=False),
TypeError: __init__() got an unexpected keyword argument 'assert_unicode'

Unit tests aren't run at all, and cinder refuses to install (because I'm
doing the db_sync in the postinst, which fails).

Help there would also be appreciated.

AFAICT, these are the only packages affected by the SQLAlchemy upgrade.

Thomas


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Campus Network Blueprint

On Jul 9, 2013, at 11:18 AM, Filipe Manco  wrote:
> 
> Hi all,
> 
> Just published a blueprint for some work I'm starting right now. I would 
> appreciate if someone can take a look and give some comments.
> 
> https://blueprints.launchpad.net/neutron/+spec/campus-network
> 
> Thank you
> 
Hi Filipe:

At first glance, there appears to be a lot of overlap with ML2 here. Have you 
looked at the ML2 blueprints, and the specific use case of multi-segment 
networks? The ML2 team is working hard to close ML2 blueprints in H2/H3, 
perhaps some of your ideas could be incorporated here?

Here is the ML2 Meeting page which has links to the blueprints we're tracking:

https://wiki.openstack.org/wiki/Meetings/ML2

Thanks,
Kyle

> 
> Filipe Manco
> http://about.me/fmanco
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] HACKING and new-style relative imports



On 07/09/2013 10:55 AM, Joe Gordon wrote:
> 
> On Tue, Jul 9, 2013 at 1:28 PM, Kieran Spear  > wrote:
> 
> Hi all,
> 
> There's a review up to make Horizon pass H304 (no relative imports):
> 
> https://review.openstack.org/#/c/35664/
> 
> Old-style relative imports are dangerous, but I can't think of a
> good enough reason to forbid new-style imports, e.g.:
> 
> from .views import IndexView
> 
> instead of
> 
> from openstack_dashboard.dashboards.project.images_and_snapshots.views \
>   import IndexView
> 
> particularly in Horizon where nesting is deep and you otherwise end
> up with many imports that won't fit into 80 characters. I think if
> we're going to make the change above the benefits would need to
> outweigh the effect on readability.
> 
> There's some discussion on the review already, but I'd like to hear
> from a more general audience. Thoughts?
> 
> 
> 
> http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Imports
> 
> "Do not use relative names in imports. Even if the module is in the same
> package, use the full package name. This helps prevent unintentionally
> importing a package twice."
> 

>From pep8:

Relative imports for intra-package imports are highly discouraged.
Always use the absolute package path for all imports. Even now that PEP
328 is fully implemented in Python 2.5, its style of explicit relative
imports is actively discouraged; absolute imports are more portable and
usually more readable.

Monty

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

On 07/09/2013 11:21 AM, Clint Byrum wrote:

>>> You may want to update your snark! MySQL has its warts which are pretty
>>> easy to take shots at, but it has been a "real" ACID compliant database
>>> for well over a decade.
>> My snark is more recently generated than that.  There are plenty of 
>> places where MySQL has fallen down.
>>
>> InnoDB support was not mandatory, and without it, MySQL was not really 
>> ACID compliant.  Using InnoDB was troublesome enough that the RHEL 6 
>> version of MySQL defaults to MyISAM.
>>
> 
> Its not so much that it was troublesome to use InnoDB as it was that
> people were used to MyISAM's few mis-features (fulltext and delayed
> inserts mostly) and needed a long warning (run up to 5.5) that the
> default was changing. Before 5.5 was released, Drizzle _completely
> removed_ MyISAM because it causes way more problems than it solves.

Let's try to avoid snarky database flame wars. There are enough of us in
this community with massive experience in each database that they are
useless to have.

MySQL and PostGres are both lovely databases that are both completely
valid choices for production. If anyone out there wants to have a beer
and attempt to convince me otherwise, you are welcome to. You will not
succeed. All snark aside, the current body of evidence for the current
state of both databases will vastly outweigh any snark. The_historical_
body of evidence for both databases has ridiculous warts, but is pointless.

That said, we, as a project, seem to have made the decision that we find
it important to support any biases you may have as a deployer by
supporting multiple database backends via SQLalchemy. However, OpenStack
is intended to be something that can be used for real large-scale
production workloads. Most large scale database apps are NOT written in
a database agnostic manner, because every database out there has
performance quirks that have to be considered, and has a different set
of tools for making things work.

How do we deal with that dichotomy? I think it might behoove us to
figure out a _sensible_ way to allow for engine-specific overrides of
sqlalchemy behavior where appropriate. The sqlaclchemy code should be
totally functional and correct, but there are clearly times when a
single well-crafted query, or taking advantage of a database-specific
extension could make a massive difference for a large deploy.

I don't think we should start peppering our code with a bunch of ifs. If
we want this, we need an understandable, sensible and repeatable
mechanism for engine-specific optimizations. Something tells me this
will not be the last time the question comes up.

Monty

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Current biggest OpenStack gate fail culprit - neutron bug #1194026

Monty Taylor wrote:
> I think having an uber-triager group comprised of the small set of gate
> watchers is not a terrible idea. Have the group, have that group be a
> member of all the other driver groups.

FWIW you actually just need to be a bug supervisor to set priority,
which in most cases means adding yourself to the PROJECT-bugs group on
LP. Some of those groups are still moderated, but most of them are open.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Campus Network Blueprint

2013-07-09 Thread Filipe Manco

Hi all,

Just published a blueprint for some work I'm starting right now. I would
appreciate if someone can take a look and give some comments.

https://blueprints.launchpad.net/neutron/+spec/campus-network

Thank you


Filipe Manco
http://about.me/fmanco
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Current biggest OpenStack gate fail culprit - neutron bug #1194026

Sean Dague wrote:
> In the corner to my left, our current largest gate reset culprit appears
> to be neutron bug #1194026 - weighing in with 62 rechecks since June
> 24th (http://status.openstack.org/rechecks/)
> 
> It has yet to be triaged by the neutron team, so is still sitting with
> undefined priority. However this seems High or Critical to me as this is
> affecting the ability of other teams to land code. It would be great if
> we could get some neutron eyes on this.

I'll raise it at the release meeting today.

> As an aside, it would probably be nice if those of us watching the gate
> we able to be in a group that could prioritize these kinds of bugs
> across projects. Currently bug prioritization is very project specific,
> which makes it easy for a bug like this to exist for 3 weeks, be our top
> gate reset issue, and not be triaged yet.

I usually try to catch those and push them up. If I fail to notice one,
please please ping me and I'll add it to the release management radar.

Regards,

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Current biggest OpenStack gate fail culprit - neutron bug #1194026



On 07/09/2013 11:41 AM, Sean Dague wrote:
> In the corner to my left, our current largest gate reset culprit appears
> to be neutron bug #1194026 - weighing in with 62 rechecks since June
> 24th (http://status.openstack.org/rechecks/)
> 
> It has yet to be triaged by the neutron team, so is still sitting with
> undefined priority. However this seems High or Critical to me as this is
> affecting the ability of other teams to land code. It would be great if
> we could get some neutron eyes on this.
> 
> As an aside, it would probably be nice if those of us watching the gate
> we able to be in a group that could prioritize these kinds of bugs
> across projects. Currently bug prioritization is very project specific,
> which makes it easy for a bug like this to exist for 3 weeks, be our top
> gate reset issue, and not be triaged yet.

I think having an uber-triager group comprised of the small set of gate
watchers is not a terrible idea. Have the group, have that group be a
member of all the other driver groups.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program Proposal: refstack

Mark McLoughlin wrote:
> On Tue, 2013-07-09 at 11:01 -0400, Monty Taylor wrote:
>>
>> I'd like to propose an official program to the TC - refstack, a program
>> for verifying interoperability between implementations via FITS testing.
>>
>> Official Title: OpenStack Interoperability
>> Initial PTL: Monty Taylor 
>> Mission Statement:
>>   Develop and maintain FITS testing and interoperability reporting for
>>   OpenStack deployments and distributions.
>> [...]
> 
> What's the progress been to date? What team of folks has been working on
> it?
> 
> I see:
> 
>   http://refstack.org/
>   https://etherpad.openstack.org/RefStackBlueprint
>   https://github.com/openstack-ops/refstack
> 
> This should be similar to how we evaluate project proposals - I'd expect
> to see a decent amount of initial progress and a fairly well solidified
> team before accepting a new program.

+1

This sounds like a good idea which is struggling to find people to
implement it. Becoming a program should not be about getting that
initial visibility push, it should be to recognize that the work an
established team has produced (or in an advanced stage of producing) is
essential to the OpenStack mission. I would have a hard time considering
something that doesn't produce anything yet to be essential...

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program Proposal: refstack



On 07/09/2013 11:23 AM, Mark McLoughlin wrote:
> Hey,
> 
> On Tue, 2013-07-09 at 11:01 -0400, Monty Taylor wrote:
>> Hey all,
>>
>> I'd like to propose an official program to the TC - refstack, a program
>> for verifying interoperability between implementations via FITS testing.
>>
>> Official Title: OpenStack Interoperability
>> Initial PTL: Monty Taylor 
>> Mission Statement:
>>   Develop and maintain FITS testing and interoperability reporting for
>>   OpenStack deployments and distributions.
>>
>> For a little background, the specific incarnation of the idea for this
>> came up at the last in-person Foundation Board meeting, and I think it's
>> a good idea. Also, it turns out there’s a reference in the logo
>> guidelines stating that any FITS defined by the TC and made available by
>> OpenStack needs to be passed before the logos can be used for a product.
>>
>> Follow on discussions were held at the summit, and a general approach of
>> basing the testing of clouds on tempest was agreed to. The general idea
>> is that for any given cloud, refstack will run tempest against the cloud
>> in a standard configuration (not re-configuring tempest on a per-cloud
>> basis) This will result in a set of passing and failing tests. That
>> output then needs to be processed and presented in a way that it can be
>> the basis of determining which elements are compliant and not. The
>> ultimate decisions around which elements would be 'required' to be a
>> compliant deployment would come back to the TC, and how that compliance
>> translates in to trademark usage goes back to the board. But it's the
>> job of refstack to be able to test the candidate cloud and report on its
>> actual capabilities.
>>
>> In conjunction with this a 'reference' deployment config is needed, so
>> that we can validate that our test is passable. At the moment I would
>> expect this to be the de-facto standard which is devstack, but that's
>> ultimately a happenstance and opportunistic config, and over time the
>> refstack program would be involved in helping to define a deployment
>> configuration or configurations that we as a community feel should
>> reasonably be expected to pass our own FITS testing.
>>
>> Expected outputs:
>>  - A service for submission of cloud endpoint information for testing
>>  - Processing of tempest output into a service compliance report
>>  - One or more deployment configurations that can, themselves, pass the
>> testing 100%
> 
> What's the progress been to date? What team of folks has been working on
> it?
> 
> I see:
> 
>   http://refstack.org/
>   https://etherpad.openstack.org/RefStackBlueprint
>   https://github.com/openstack-ops/refstack
> 
> This should be similar to how we evaluate project proposals - I'd expect
> to see a decent amount of initial progress and a fairly well solidified
> team before accepting a new program.

Excellent call... we've been mostly in planning stages so far, so how
about I withdraw the formal application for now, let us get some things
up for you to look at, then come back in a few weeks.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program Proposal: refstack



On 07/09/2013 11:24 AM, Joshua McKenty wrote:
> Monty, you missed a couple of elements of the original program
> definition - the mappings from the tempest based "scorecard" to specific
> versions of the API docs, and the availability of the tested and
> verified endpoints to "tools ecosystem" developers.

Yes, great catch.

> Also, I believe the definition of "compliance" rests with the Board and
> not the TC, despite some debate as to whether we're capable of handling it.

I think it's a collaboration. I think the board owns the definition of
compliance, but I think that the TC has got to own the technical
implementation of that definition, both in the tooling but also in the
technical assessment of the results of tests. I don't think we need to
get legalistic with this yet... let's get something in front of people,
then we can battle on who gets to own looking at test result matrixes. :)

> And I'll arm-wrestle you for the PTL slot ;)

We should do that on a web-case...

> PS - for the purposes of FITS, the "candidate cloud" can be either a
> public cloud, or a test environment for private cloud software.

Yes, I believe someone would need to be able to run a copy of this
locally as well.

> On Jul 9, 2013 8:03 AM, "Monty Taylor"  > wrote:
> 
> Hey all,
> 
> I'd like to propose an official program to the TC - refstack, a program
> for verifying interoperability between implementations via FITS testing.
> 
> Official Title: OpenStack Interoperability
> Initial PTL: Monty Taylor  >
> Mission Statement:
>   Develop and maintain FITS testing and interoperability reporting for
>   OpenStack deployments and distributions.
> 
> For a little background, the specific incarnation of the idea for this
> came up at the last in-person Foundation Board meeting, and I think it's
> a good idea. Also, it turns out there’s a reference in the logo
> guidelines stating that any FITS defined by the TC and made available by
> OpenStack needs to be passed before the logos can be used for a product.
> 
> Follow on discussions were held at the summit, and a general approach of
> basing the testing of clouds on tempest was agreed to. The general idea
> is that for any given cloud, refstack will run tempest against the cloud
> in a standard configuration (not re-configuring tempest on a per-cloud
> basis) This will result in a set of passing and failing tests. That
> output then needs to be processed and presented in a way that it can be
> the basis of determining which elements are compliant and not. The
> ultimate decisions around which elements would be 'required' to be a
> compliant deployment would come back to the TC, and how that compliance
> translates in to trademark usage goes back to the board. But it's the
> job of refstack to be able to test the candidate cloud and report on its
> actual capabilities.
> 
> In conjunction with this a 'reference' deployment config is needed, so
> that we can validate that our test is passable. At the moment I would
> expect this to be the de-facto standard which is devstack, but that's
> ultimately a happenstance and opportunistic config, and over time the
> refstack program would be involved in helping to define a deployment
> configuration or configurations that we as a community feel should
> reasonably be expected to pass our own FITS testing.
> 
> Expected outputs:
>  - A service for submission of cloud endpoint information for testing
>  - Processing of tempest output into a service compliance report
>  - One or more deployment configurations that can, themselves, pass the
> testing 100%
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Heat] awareness of nova cells, availability zones etc.

2013-07-09 Thread Eoghan Glynn


Hi Folks,

Just wondering to what extent Heat deployment policies can be made
nova cell/AZ-aware?

For example, one could envisage a situation in which it would sense
for an autoscaling group to have cell-affinity, but also have AZ-
anti-affinity within the cell for resiliance against datacenter
outages.

Are those kinds of policies currently supported by Heat?

Cheers,
Eoghan

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Keystone] How to write unit tests for db methods?

2013-07-09 Thread Akshat Kakkar

How to write unit tests in keystone for the methods which are directly calling 
the backend db? I understand that for testing purpose it should be a *fake db*, 
but how to do that in keystone?
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Hyper-V meeting agenda

2013-07-09 Thread Peter Pouliot

Hi All,

It's been a while since we've all been able to meet.  This week will primarily 
be used to catch up on open items and get an overall status of where things are.

p

Peter J. Pouliot, CISSP
Senior SDET, OpenStack

Microsoft
New England Research & Development Center
One Memorial Drive,Cambridge, MA 02142
ppoul...@microsoft.com | Tel: +1(857) 453 6436

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Current biggest OpenStack gate fail culprit - neutron bug #1194026

2013-07-09 Thread Sean Dague

In the corner to my left, our current largest gate reset culprit appears 
to be neutron bug #1194026 - weighing in with 62 rechecks since June 
24th (http://status.openstack.org/rechecks/)


It has yet to be triaged by the neutron team, so is still sitting with 
undefined priority. However this seems High or Critical to me as this is 
affecting the ability of other teams to land code. It would be great if 
we could get some neutron eyes on this.


As an aside, it would probably be nice if those of us watching the gate 
we able to be in a group that could prioritize these kinds of bugs 
across projects. Currently bug prioritization is very project specific, 
which makes it easy for a bug like this to exist for 3 weeks, be our top 
gate reset issue, and not be triaged yet.


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo

On Tue, 2013-07-09 at 11:39 -0400, Doug Hellmann wrote:
> 
> 
> 
> On Tue, Jul 9, 2013 at 6:11 AM, Mark McLoughlin  wrote:
> Hey
> 
> The mission statement is what we've been using for a while. The
> "official title" is new.
> 
>   Official Title: OpenStack Common Libraries
>   PTL: Mark McLoughlin 
>   Mission Statement:
> To produce a set of python libraries containing infrastructure 
> code
> shared by OpenStack projects. The APIs provided by these libraries
> should be high quality, stable, consistent and generally 
> applicable.
> 
> 
> +1
> 
> 
> What sort of code would not be "infrastructure" but would be "shared
> by OpenStack projects"? Is the "infrastructure" redundant?

Simply saying "code" isn't enterprisey enough

(Point taken, removed :)

Mark.




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo

2013-07-09 Thread Doug Hellmann

On Tue, Jul 9, 2013 at 6:11 AM, Mark McLoughlin  wrote:

> Hey
>
> The mission statement is what we've been using for a while. The
> "official title" is new.
>
>   Official Title: OpenStack Common Libraries
>   PTL: Mark McLoughlin 
>   Mission Statement:
> To produce a set of python libraries containing infrastructure code
> shared by OpenStack projects. The APIs provided by these libraries
> should be high quality, stable, consistent and generally applicable.
>

+1

What sort of code would not be "infrastructure" but would be "shared by
OpenStack projects"? Is the "infrastructure" redundant?

Doug


>
> I did consider explicitly mentioning technical debt with something like:
>
>   Mission Statement:
> To tackle copy-and-paste technical debt in OpenStack by producing a
> set of python libraries containing infrastructure code shared by
> OpenStack projects. The APIs provided by these libraries should be
> high quality, stable, consistent and generally applicable.
>
> But for wholly new code, that sounds like you need to introduce
> copy-and-paste technical debt before it can be considered in scope for
> Oslo :)
>
> Cheers,
> Mark.
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

2013-07-09 Thread Dolph Mathews

On Tuesday, July 9, 2013, Clint Byrum wrote:

> Excerpts from Adam Young's message of 2013-07-09 06:19:19 -0700:
> > On 07/08/2013 11:06 PM, Clint Byrum wrote:
> > > Excerpts from Adam Young's message of 2013-07-08 17:32:30 -0700:
> > >> On 07/08/2013 04:35 PM, Clint Byrum wrote:
> > >>> Excerpts from Adam Young's message of 2013-07-08 13:18:55 -0700:
> >  On 07/01/2013 01:35 PM, Clint Byrum wrote:
> > > The way the new keystone-manage command "token_flush" works right
> now
> > > is quite broken by MySQL and InnoDB's gap locking behavior:
> > >
> > > https://bugs.launchpad.net/1188378
> > >
> > > Presumably other SQL databases like PostgreSQL will have similar
> problems
> > > with doing massive deletes, but I am less familiar with them.
> > >
> > > I am trying to solve this in keystone, and my first attempt is
> here:
> > >
> > > https://review.openstack.org/#/c/32044/
> > >
> > > However, MySQL does not support using "LIMIT" in a sub-query that
> > > is feeding an IN() clause, so that approach will not work.
> Likewise,
> > > sqlalchemy does not support the MySQL specific extension to DELETE
> which
> > > allows it to have a LIMIT clause.
> > >
> > > Now, I can do some hacky things, like just deleting all of the
> expired
> > > tokens from the oldest single second, but that could also
> potentially
> > > be millions of tokens, and thus, millions of gaps to lock.
> > >
> > > So, there is just not one way to work for all databases, and we
> have to
> > > have a special mode for MySQL.
> > >
> > > I was wondering if anybody has suggestions and/or examples of how
> to do
> > > that with sqlalchemy.
> > >
> > > ___
> > > OpenStack-dev mailing list
> > > OpenStack-dev@lists.openstack.org 
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >  In general, if you have millions of roles, you need a real
> database.  I
> >  would not try to work through SQL Alchemy for this. Instead, you
> >  probably just want to make sure that the token_flush is run fairly
> >  regularly on your database.
> > 
> > >>> I'm not sure I understand you.
> > >>>
> > >>> * I am asking about millions of tokens, not roles.
> > >> Heh, I mean Rows, and somehow type roles.
> > >>
> > >>> * I am asking about MySQL.. presumably a "real" database.
> > >>I have to admit I am a bit of a Postgresql Bigot. I don't really
> > >> consider MySQL a real database, althought it has improved a lot over
> the
> > >> years.  I am not up to speed on"InnoDB's gap locking behavior" but it
> is
> > >> not something I would expect to be a problem in Postgresql.
> > >>
> > > You may want to update your snark! MySQL has its warts which are pretty
> > > easy to take shots at, but it has been a "real" ACID compliant database
> > > for well over a decade.
> > My snark is more recently generated than that.  There are plenty of
> > places where MySQL has fallen down.
> >
> > InnoDB support was not mandatory, and without it, MySQL was not really
> > ACID compliant.  Using InnoDB was troublesome enough that the RHEL 6
> > version of MySQL defaults to MyISAM.
> >
>
> Its not so much that it was troublesome to use InnoDB as it was that
> people were used to MyISAM's few mis-features (fulltext and delayed
> inserts mostly) and needed a long warning (run up to 5.5) that the
> default was changing. Before 5.5 was released, Drizzle _completely
> removed_ MyISAM because it causes way more problems than it solves.
>
> > >
> > > Please have a read at how InnoDB's isolation level handling works
> > > [1]. You can compare it to Postgres [2]. InnoDB's default isolation
> > > level, repeatable read, has a locking behavior that is particularly
> > > ugly for indexes like 'token.valid'. Put simply, you can't add more
> > > valid tokens until you're done deleting any valid expired tokens (if
> > > you mention valid in the delete, which flush_tokens does not).
> > >
> > > This is because token.valid == 1 may end up locked during the entire
> > > DELETE. The same could be true for expires too, even the primary key,
> > > though that is typically not locked with ranges.
> > >
> > > Yes, Postgres's behaviors are much more desirable here, as inserts
> almost
> > > go unmentioned in the postgres description of transaction isolation.
> But
> > > there are well documented ways of working around this weirdness. I had
> > > forgotten that one way is to use READ UNCOMMITTED. That should avoid
> > > any of the large range locks and work with MySQL, Postgres, and Sqlite.
> >
> >
> > I think the real solution is for us to stop writing tokens to the
> > Database.  If we are using PKI tokens, there is no need for a database
> > entry for valid tokens, only for revoked tokens.
> >
> > That makes this whole thing into a non-problem.
>
> Yes please! Is this something that is in developmen

Re: [openstack-dev] Ceilometer XSD

2013-07-09 Thread Doug Hellmann

The instructions for getting set up to contribute to OpenStack in general
are at https://wiki.openstack.org/wiki/How_To_Contribute (maybe you've
already done that, I'm not sure).

As far as this specific change goes, I would have to understand more about
what it is that you want added to give specific advice. If we don't have
the tools to make the XSD, I'm not sure how we will ensure it is kept up to
date with changes to the API.

Doug


On Mon, Jul 8, 2013 at 1:33 PM, Jobin Raju George wrote:

> Hey, all!
>
> I have prepared the XSD for ceilometer and would like to contribute it to
> the repositories on github. Can somebody help me out with the process to do
> this?
>
>
> On Mon, Jul 8, 2013 at 2:19 PM, Jobin Raju George wrote:
>
>> Ok, that's fine. I am currently investigating into it. Lets see if I get
>> any clues. Thanks for you time!
>>
>>
>> On Mon, Jul 8, 2013 at 1:32 PM, Julien Danjou  wrote:
>>
>>> On Sat, Jul 06 2013, Jobin Raju George wrote:
>>>
>>> > I am trying to use the meters provided by ceilometer to extract usage
>>> > values from the VM's deployed using openstack.
>>> >
>>> > However, in order to programmatically do this I need the XSD files for
>>> > ceilometer. I tried googling them, posting them on forums and even on
>>> > launchpad but have not received any response, i would be great if you
>>> could
>>> > provide me with the source/link/file of the XSD's.
>>>
>>> The XML API is provided automatically via WSME¹, and I've no idea if it
>>> provides any XSD automatically generated or something like that.
>>>
>>> ¹  https://pypi.python.org/pypi/WSME
>>>
>>> --
>>> Julien Danjou
>>> /* Free Software hacker * freelance consultant
>>>http://julien.danjou.info */
>>>
>>
>>
>>
>> --
>>
>>  Thanks and regards,
>>
>> Jobin Raju George
>>
>> Third Year, Information Technology
>>
>> College of Engineering Pune
>>
>> Alternate e-mail: georgejr10...@coep.ac.in
>>
>>
>
>
> --
>
> Thanks and regards,
>
> Jobin Raju George
>
> Third Year, Information Technology
>
> College of Engineering Pune
>
> Alternate e-mail: georgejr10...@coep.ac.in
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Program Proposal: refstack

Hey all,

I'd like to propose an official program to the TC - refstack, a program
for verifying interoperability between implementations via FITS testing.

Official Title: OpenStack Interoperability
Initial PTL: Monty Taylor 
Mission Statement:
  Develop and maintain FITS testing and interoperability reporting for
  OpenStack deployments and distributions.

For a little background, the specific incarnation of the idea for this
came up at the last in-person Foundation Board meeting, and I think it's
a good idea. Also, it turns out there’s a reference in the logo
guidelines stating that any FITS defined by the TC and made available by
OpenStack needs to be passed before the logos can be used for a product.

Follow on discussions were held at the summit, and a general approach of
basing the testing of clouds on tempest was agreed to. The general idea
is that for any given cloud, refstack will run tempest against the cloud
in a standard configuration (not re-configuring tempest on a per-cloud
basis) This will result in a set of passing and failing tests. That
output then needs to be processed and presented in a way that it can be
the basis of determining which elements are compliant and not. The
ultimate decisions around which elements would be 'required' to be a
compliant deployment would come back to the TC, and how that compliance
translates in to trademark usage goes back to the board. But it's the
job of refstack to be able to test the candidate cloud and report on its
actual capabilities.

In conjunction with this a 'reference' deployment config is needed, so
that we can validate that our test is passable. At the moment I would
expect this to be the de-facto standard which is devstack, but that's
ultimately a happenstance and opportunistic config, and over time the
refstack program would be involved in helping to define a deployment
configuration or configurations that we as a community feel should
reasonably be expected to pass our own FITS testing.

Expected outputs:
 - A service for submission of cloud endpoint information for testing
 - Processing of tempest output into a service compliance report
 - One or more deployment configurations that can, themselves, pass the
testing 100%

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program Proposal: refstack

2013-07-09 Thread Walls, Jeffrey Joel (HP Converged Cloud - Cloud OS)

Hey,

On Tue, 2013-07-09 at 11:01 -0400, Monty Taylor wrote:
> Hey all,
> 
> I'd like to propose an official program to the TC - refstack, a program
> for verifying interoperability between implementations via FITS testing.
> 
> Official Title: OpenStack Interoperability
> Initial PTL: Monty Taylor 
> Mission Statement:
>   Develop and maintain FITS testing and interoperability reporting for
>   OpenStack deployments and distributions.
> 
> For a little background, the specific incarnation of the idea for this
> came up at the last in-person Foundation Board meeting, and I think it's
> a good idea. Also, it turns out there’s a reference in the logo
> guidelines stating that any FITS defined by the TC and made available by
> OpenStack needs to be passed before the logos can be used for a product.
> 
> Follow on discussions were held at the summit, and a general approach of
> basing the testing of clouds on tempest was agreed to. The general idea
> is that for any given cloud, refstack will run tempest against the cloud
> in a standard configuration (not re-configuring tempest on a per-cloud
> basis) This will result in a set of passing and failing tests. That
> output then needs to be processed and presented in a way that it can be
> the basis of determining which elements are compliant and not. The
> ultimate decisions around which elements would be 'required' to be a
> compliant deployment would come back to the TC, and how that compliance
> translates in to trademark usage goes back to the board. But it's the
> job of refstack to be able to test the candidate cloud and report on its
> actual capabilities.
> 
> In conjunction with this a 'reference' deployment config is needed, so
> that we can validate that our test is passable. At the moment I would
> expect this to be the de-facto standard which is devstack, but that's
> ultimately a happenstance and opportunistic config, and over time the
> refstack program would be involved in helping to define a deployment
> configuration or configurations that we as a community feel should
> reasonably be expected to pass our own FITS testing.
> 
> Expected outputs:
>  - A service for submission of cloud endpoint information for testing
>  - Processing of tempest output into a service compliance report
>  - One or more deployment configurations that can, themselves, pass the
> testing 100%

What's the progress been to date? What team of folks has been working on
it?

I see:

  http://refstack.org/
  https://etherpad.openstack.org/RefStackBlueprint
  https://github.com/openstack-ops/refstack

This should be similar to how we evaluate project proposals - I'd expect
to see a decent amount of initial progress and a fairly well solidified
team before accepting a new program.

Cheers,
Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program Proposal: refstack

2013-07-09 Thread Joshua McKenty

Monty, you missed a couple of elements of the original program definition -
the mappings from the tempest based "scorecard" to specific versions of the
API docs, and the availability of the tested and verified endpoints to
"tools ecosystem" developers.

Also, I believe the definition of "compliance" rests with the Board and not
the TC, despite some debate as to whether we're capable of handling it.

And I'll arm-wrestle you for the PTL slot ;)

PS - for the purposes of FITS, the "candidate cloud" can be either a public
cloud, or a test environment for private cloud software.
On Jul 9, 2013 8:03 AM, "Monty Taylor"  wrote:

> Hey all,
>
> I'd like to propose an official program to the TC - refstack, a program
> for verifying interoperability between implementations via FITS testing.
>
> Official Title: OpenStack Interoperability
> Initial PTL: Monty Taylor 
> Mission Statement:
>   Develop and maintain FITS testing and interoperability reporting for
>   OpenStack deployments and distributions.
>
> For a little background, the specific incarnation of the idea for this
> came up at the last in-person Foundation Board meeting, and I think it's
> a good idea. Also, it turns out there’s a reference in the logo
> guidelines stating that any FITS defined by the TC and made available by
> OpenStack needs to be passed before the logos can be used for a product.
>
> Follow on discussions were held at the summit, and a general approach of
> basing the testing of clouds on tempest was agreed to. The general idea
> is that for any given cloud, refstack will run tempest against the cloud
> in a standard configuration (not re-configuring tempest on a per-cloud
> basis) This will result in a set of passing and failing tests. That
> output then needs to be processed and presented in a way that it can be
> the basis of determining which elements are compliant and not. The
> ultimate decisions around which elements would be 'required' to be a
> compliant deployment would come back to the TC, and how that compliance
> translates in to trademark usage goes back to the board. But it's the
> job of refstack to be able to test the candidate cloud and report on its
> actual capabilities.
>
> In conjunction with this a 'reference' deployment config is needed, so
> that we can validate that our test is passable. At the moment I would
> expect this to be the de-facto standard which is devstack, but that's
> ultimately a happenstance and opportunistic config, and over time the
> refstack program would be involved in helping to define a deployment
> configuration or configurations that we as a community feel should
> reasonably be expected to pass our own FITS testing.
>
> Expected outputs:
>  - A service for submission of cloud endpoint information for testing
>  - Processing of tempest output into a service compliance report
>  - One or more deployment configurations that can, themselves, pass the
> testing 100%
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

2013-07-09 Thread Clint Byrum

Excerpts from Adam Young's message of 2013-07-09 06:19:19 -0700:
> On 07/08/2013 11:06 PM, Clint Byrum wrote:
> > Excerpts from Adam Young's message of 2013-07-08 17:32:30 -0700:
> >> On 07/08/2013 04:35 PM, Clint Byrum wrote:
> >>> Excerpts from Adam Young's message of 2013-07-08 13:18:55 -0700:
>  On 07/01/2013 01:35 PM, Clint Byrum wrote:
> > The way the new keystone-manage command "token_flush" works right now
> > is quite broken by MySQL and InnoDB's gap locking behavior:
> >
> > https://bugs.launchpad.net/1188378
> >
> > Presumably other SQL databases like PostgreSQL will have similar 
> > problems
> > with doing massive deletes, but I am less familiar with them.
> >
> > I am trying to solve this in keystone, and my first attempt is here:
> >
> > https://review.openstack.org/#/c/32044/
> >
> > However, MySQL does not support using "LIMIT" in a sub-query that
> > is feeding an IN() clause, so that approach will not work. Likewise,
> > sqlalchemy does not support the MySQL specific extension to DELETE which
> > allows it to have a LIMIT clause.
> >
> > Now, I can do some hacky things, like just deleting all of the expired
> > tokens from the oldest single second, but that could also potentially
> > be millions of tokens, and thus, millions of gaps to lock.
> >
> > So, there is just not one way to work for all databases, and we have to
> > have a special mode for MySQL.
> >
> > I was wondering if anybody has suggestions and/or examples of how to do
> > that with sqlalchemy.
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>  In general, if you have millions of roles, you need a real database.  I
>  would not try to work through SQL Alchemy for this. Instead, you
>  probably just want to make sure that the token_flush is run fairly
>  regularly on your database.
> 
> >>> I'm not sure I understand you.
> >>>
> >>> * I am asking about millions of tokens, not roles.
> >> Heh, I mean Rows, and somehow type roles.
> >>
> >>> * I am asking about MySQL.. presumably a "real" database.
> >>I have to admit I am a bit of a Postgresql Bigot. I don't really
> >> consider MySQL a real database, althought it has improved a lot over the
> >> years.  I am not up to speed on"InnoDB's gap locking behavior" but it is
> >> not something I would expect to be a problem in Postgresql.
> >>
> > You may want to update your snark! MySQL has its warts which are pretty
> > easy to take shots at, but it has been a "real" ACID compliant database
> > for well over a decade.
> My snark is more recently generated than that.  There are plenty of 
> places where MySQL has fallen down.
> 
> InnoDB support was not mandatory, and without it, MySQL was not really 
> ACID compliant.  Using InnoDB was troublesome enough that the RHEL 6 
> version of MySQL defaults to MyISAM.
> 

Its not so much that it was troublesome to use InnoDB as it was that
people were used to MyISAM's few mis-features (fulltext and delayed
inserts mostly) and needed a long warning (run up to 5.5) that the
default was changing. Before 5.5 was released, Drizzle _completely
removed_ MyISAM because it causes way more problems than it solves.

> >
> > Please have a read at how InnoDB's isolation level handling works
> > [1]. You can compare it to Postgres [2]. InnoDB's default isolation
> > level, repeatable read, has a locking behavior that is particularly
> > ugly for indexes like 'token.valid'. Put simply, you can't add more
> > valid tokens until you're done deleting any valid expired tokens (if
> > you mention valid in the delete, which flush_tokens does not).
> >
> > This is because token.valid == 1 may end up locked during the entire
> > DELETE. The same could be true for expires too, even the primary key,
> > though that is typically not locked with ranges.
> >
> > Yes, Postgres's behaviors are much more desirable here, as inserts almost
> > go unmentioned in the postgres description of transaction isolation. But
> > there are well documented ways of working around this weirdness. I had
> > forgotten that one way is to use READ UNCOMMITTED. That should avoid
> > any of the large range locks and work with MySQL, Postgres, and Sqlite.
> 
> 
> I think the real solution is for us to stop writing tokens to the 
> Database.  If we are using PKI tokens, there is no need for a database 
> entry for valid tokens, only for revoked tokens.
> 
> That makes this whole thing into a non-problem.

Yes please! Is this something that is in development, or at least recorded
in a bug that someone like me can grab? I would love for that to be the
default behavior.

Of course, that would not solve the issue for those who want to audit
tokens (is this a large contingent of users?). In fact the f

Re: [openstack-dev] HACKING and new-style relative imports

2013-07-09 Thread Joe Gordon

On Tue, Jul 9, 2013 at 1:28 PM, Kieran Spear  wrote:

> Hi all,
>
> There's a review up to make Horizon pass H304 (no relative imports):
>
> https://review.openstack.org/#/c/35664/
>
> Old-style relative imports are dangerous, but I can't think of a good
> enough reason to forbid new-style imports, e.g.:
>
> from .views import IndexView
>
> instead of
>
> from openstack_dashboard.dashboards.project.images_and_snapshots.views \
>   import IndexView
>
> particularly in Horizon where nesting is deep and you otherwise end up
> with many imports that won't fit into 80 characters. I think if we're going
> to make the change above the benefits would need to outweigh the effect on
> readability.
>
> There's some discussion on the review already, but I'd like to hear from a
> more general audience. Thoughts?
>


http://google-styleguide.googlecode.com/svn/trunk/pyguide.html#Imports

"Do not use relative names in imports. Even if the module is in the same
package, use the full package name. This helps prevent unintentionally
importing a package twice."



>
> Cheers,
> Kieran
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Horizon] Navigation UX Enhancements - Collecting Issues

One issue I have is that the panels are physically grouped and it’s very 
difficult to logically re-group them.  For example, the Project dashboard 
explicitly lists the panel groups and the order of those panel groups.  Within 
each panel group, the individual panels are also explicitly listed.  What I 
would like to do is arrange the panels more logically without affecting the 
physical structure of the files or the order in the panel specification.

You could think of “Deployment” as a “section” under the Project dashboard tab. 
 In this “section”, I’d want to see things related to the actual deployment of 
virtual machines (e.g., Instances, Snapshots, Networks, Routers, etc).  I was 
beginning to tackle this in our code base and was planning to use some sort of 
accordion-type widget.  My thinking was that there would be “a few” (probably 
no more than 4) “sections” under the Project tab.  Each “section” would have 
elements within it that logically mapped to that section.

I think this is a great discussion and I’m very interested to hear where others 
are headed with their thinking, so thank you for getting it started!

Jeff

From: Jaromir Coufal [mailto:jcou...@redhat.com]
Sent: Tuesday, July 09, 2013 6:38 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev] [Horizon] Navigation UX Enhancements - Collecting 
Issues

Hi everybody,

in UX community group on G+ popped out a need for enhancing user experience of 
main navigation, because there are spreading out various issues .

There is already created a BP for this: 
https://blueprints.launchpad.net/horizon/+spec/navigation-enhancement

Toshi had great idea to start discussion about navigation issues on mailing 
list.

So I'd like to ask all of you, if you have some issues with navigation, what 
are the issues you are dealing with? I'd like to gather as much feedback as 
possible, so we can design the best solution which covers most of the cases. 
Issues will be listed in BP and I will try to come out with design proposals 
which hopefully will help all of you.

Examples are following:
* Navigation is not scaling for more dashboards (Project, Admin, ...)
* Each dashboard might contain different hierarchy (number of levels)

What problems do you experience with navigation?

Thanks all for contributing
-- Jarda
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo

2013-07-09 Thread Flavio Percoco


On 09/07/13 11:33 +0100, Mark McLoughlin wrote:

On Tue, 2013-07-09 at 11:21 +0100, Martyn Taylor wrote:

On 09/07/13 11:11, Mark McLoughlin wrote:
Is it worth adding some emphasis on documentation in there somewhere?
Specifically documentation for using the libraries and frameworks
offered by Oslo in OpenStack.  Maybe:

   Mission Statement:
 To produce a set of python libraries containing infrastructure code
 shared by OpenStack projects. The APIs provided by these libraries
 should be high quality, stable, consistent, well documented and
 generally applicable.


I would have seen that as implicit, but no problem including it.



"

+1 for the description!

Cheers,
FF


--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift debugging / performance - large latencies seen.

2013-07-09 Thread Chmouel Boudjnah

Hi Tim,

it's the first link on google for "apache keystone" :

http://docs.openstack.org/developer/keystone/apache-httpd.html

 we probably want to move this to openstack-operators@

Chmouel.

On Tue, Jul 9, 2013 at 3:31 PM, Snider, Tim  wrote:
> Thanks for the hint. Is there documentation on how to run keystone using
> apache and multiple processes?
>
> I’m pretty raw with python, ruby, apache …
>
> Thx.
>
>
>
> From: Chmouel Boudjnah [mailto:chmo...@enovance.com]
> Sent: Tuesday, July 09, 2013 8:22 AM
> To: OpenStack Development Mailing List
> Subject: Re: [openstack-dev] Swift debugging / performance - large latencies
> seen.
>
>
>
> On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim  wrote:
>
> I have 2 openstack clusters running the Folsom release with multiple Swift
> nodes. I also have a small setup that is running only Swift with a single
> node.  I’m noticing very large Swift I/O latencies (seconds long) on the
> openstack clusters – ssbench output snippet is below. Performance is
> approximately identical on the openstack clusters. The Swift only cluster
> performs much better.
>
>
> Keystone performance can be pretty awful unless you are using something else
> than the default WSGI container configuration (single process eventlet I
> think). I would suggest you try to run it under apache with multiple
> process.
>
> See the dicussion at last summit about Keystone performance here :
>
> https://etherpad.openstack.org/havana-keystone-performance
>
> Chmouel.
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift debugging / performance - large latencies seen.

Thanks for the hint. Is there documentation on how to run keystone using apache 
and multiple processes?
I'm pretty raw with python, ruby, apache ...
Thx.

From: Chmouel Boudjnah [mailto:chmo...@enovance.com]
Sent: Tuesday, July 09, 2013 8:22 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Swift debugging / performance - large latencies 
seen.

On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim 
mailto:tim.sni...@netapp.com>> wrote:
I have 2 openstack clusters running the Folsom release with multiple Swift 
nodes. I also have a small setup that is running only Swift with a single node. 
 I'm noticing very large Swift I/O latencies (seconds long) on the openstack 
clusters - ssbench output snippet is below. Performance is approximately 
identical on the openstack clusters. The Swift only cluster performs much 
better.

Keystone performance can be pretty awful unless you are using something else 
than the default WSGI container configuration (single process eventlet I 
think). I would suggest you try to run it under apache with multiple process.

See the dicussion at last summit about Keystone performance here :

https://etherpad.openstack.org/havana-keystone-performance

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift debugging / performance - large latencies seen.

2013-07-09 Thread Chmouel Boudjnah

On Tue, Jul 9, 2013 at 2:40 PM, Snider, Tim  wrote:

> I have 2 openstack clusters running the Folsom release with multiple Swift
> nodes. I also have a small setup that is running only Swift with a single
> node.  I’m noticing very large Swift I/O latencies (seconds long) on the
> openstack clusters – ssbench output snippet is below. Performance is
> approximately identical on the openstack clusters. The Swift only cluster
> performs much better. 
>

Keystone performance can be pretty awful unless you are using something
else than the default WSGI container configuration (single process eventlet
I think). I would suggest you try to run it under apache with multiple
process.

See the dicussion at last summit about Keystone performance here :

https://etherpad.openstack.org/havana-keystone-performance

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Best way to do something MySQL-specific?

2013-07-09 Thread Adam Young


On 07/08/2013 11:06 PM, Clint Byrum wrote:

Excerpts from Adam Young's message of 2013-07-08 17:32:30 -0700:

On 07/08/2013 04:35 PM, Clint Byrum wrote:

Excerpts from Adam Young's message of 2013-07-08 13:18:55 -0700:

On 07/01/2013 01:35 PM, Clint Byrum wrote:

The way the new keystone-manage command "token_flush" works right now
is quite broken by MySQL and InnoDB's gap locking behavior:

https://bugs.launchpad.net/1188378

Presumably other SQL databases like PostgreSQL will have similar problems
with doing massive deletes, but I am less familiar with them.

I am trying to solve this in keystone, and my first attempt is here:

https://review.openstack.org/#/c/32044/

However, MySQL does not support using "LIMIT" in a sub-query that
is feeding an IN() clause, so that approach will not work. Likewise,
sqlalchemy does not support the MySQL specific extension to DELETE which
allows it to have a LIMIT clause.

Now, I can do some hacky things, like just deleting all of the expired
tokens from the oldest single second, but that could also potentially
be millions of tokens, and thus, millions of gaps to lock.

So, there is just not one way to work for all databases, and we have to
have a special mode for MySQL.

I was wondering if anybody has suggestions and/or examples of how to do
that with sqlalchemy.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

In general, if you have millions of roles, you need a real database.  I
would not try to work through SQL Alchemy for this. Instead, you
probably just want to make sure that the token_flush is run fairly
regularly on your database.


I'm not sure I understand you.

* I am asking about millions of tokens, not roles.

Heh, I mean Rows, and somehow type roles.


* I am asking about MySQL.. presumably a "real" database.

   I have to admit I am a bit of a Postgresql Bigot. I don't really
consider MySQL a real database, althought it has improved a lot over the
years.  I am not up to speed on"InnoDB's gap locking behavior" but it is
not something I would expect to be a problem in Postgresql.


You may want to update your snark! MySQL has its warts which are pretty
easy to take shots at, but it has been a "real" ACID compliant database
for well over a decade.
My snark is more recently generated than that.  There are plenty of 
places where MySQL has fallen down.


InnoDB support was not mandatory, and without it, MySQL was not really 
ACID compliant.  Using InnoDB was troublesome enough that the RHEL 6 
version of MySQL defaults to MyISAM.





Please have a read at how InnoDB's isolation level handling works
[1]. You can compare it to Postgres [2]. InnoDB's default isolation
level, repeatable read, has a locking behavior that is particularly
ugly for indexes like 'token.valid'. Put simply, you can't add more
valid tokens until you're done deleting any valid expired tokens (if
you mention valid in the delete, which flush_tokens does not).

This is because token.valid == 1 may end up locked during the entire
DELETE. The same could be true for expires too, even the primary key,
though that is typically not locked with ranges.

Yes, Postgres's behaviors are much more desirable here, as inserts almost
go unmentioned in the postgres description of transaction isolation. But
there are well documented ways of working around this weirdness. I had
forgotten that one way is to use READ UNCOMMITTED. That should avoid
any of the large range locks and work with MySQL, Postgres, and Sqlite.



I think the real solution is for us to stop writing tokens to the 
Database.  If we are using PKI tokens, there is no need for a database 
entry for valid tokens, only for revoked tokens.


That makes this whole thing into a non-problem.




[1] 
http://dev.mysql.com/doc/refman/5.5/en/set-transaction.html#isolevel_repeatable-read
[2] 
http://www.postgresql.org/docs/9.1/static/transaction-iso.html#MVCC-SERIALIZABILITY


* In the bug, I am suggesting that running token_flush once every
second will be _a disaster_ on a busy site with MySQL because of
the gap locking behavior in InnoDB. We need to delete a small number
per transaction.

once every second would be strange indeed.  I would think maybe once
every five minutes or so.  Schedule your clean up IAW your deployment
and usage.

Deleting a chunk of tokens in bulk would be preferable to doing client
side iteration, I can;t see how that would not be the case.


Once every second or once every 5 minutes, the locks pile up the longer
the DELETE takes.

Using pt-archiver, the locks span just a few hundred rows at a time, and
we can also be "gentle" with the server, sleeping a bit between deletes,
which allows us to not waste precious OLTP time on archiving/deleting.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/

Re: [openstack-dev] Program description for Oslo

2013-07-09 Thread Davanum Srinivas

+1

On Tue, Jul 9, 2013 at 8:07 AM, Thierry Carrez  wrote:
> Michael Still wrote:
>> On Tue, Jul 9, 2013 at 8:11 PM, Mark McLoughlin  wrote:
>>> The mission statement is what we've been using for a while. The
>>> "official title" is new.
>>>
>>>   Official Title: OpenStack Common Libraries
>>>   PTL: Mark McLoughlin 
>>>   Mission Statement:
>>> To produce a set of python libraries containing infrastructure code
>>> shared by OpenStack projects. The APIs provided by these libraries
>>> should be high quality, stable, consistent and generally applicable.
>>
>> This description looks good to me.
>
> +1
>
> --
> Thierry Carrez (ttx)
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: http://davanum.wordpress.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo

On Tue, 2013-07-09 at 08:41 -0400, Jonathan Bryce wrote:
> Any reason for not just calling it "OpenStack Commons"?

Yes - we're very much focused on libraries of shared code. When we used
the name "OpenStack Common" previously, people made all sorts of weird
assumptions about what could go in there.

Cheers,
Mark.

> Mark McLoughlin  wrote:
> 
> >Hey
> >
> >The mission statement is what we've been using for a while. The
> >"official title" is new.
> >
> >  Official Title: OpenStack Common Libraries
> >  PTL: Mark McLoughlin 
> >  Mission Statement:
> >To produce a set of python libraries containing infrastructure code 
> >shared by OpenStack projects. The APIs provided by these libraries
> >should be high quality, stable, consistent and generally applicable.
> >
> >I did consider explicitly mentioning technical debt with something like:
> >
> >  Mission Statement:
> >To tackle copy-and-paste technical debt in OpenStack by producing a
> >set of python libraries containing infrastructure code shared by
> >OpenStack projects. The APIs provided by these libraries should be
> >high quality, stable, consistent and generally applicable.
> >
> >But for wholly new code, that sounds like you need to introduce
> >copy-and-paste technical debt before it can be considered in scope for
> >Oslo :)
> >
> >Cheers,
> >Mark.
> >
> >
> >___
> >OpenStack-dev mailing list
> >OpenStack-dev@lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo

2013-07-09 Thread Jonathan Bryce

Any reason for not just calling it "OpenStack Commons"?

Mark McLoughlin  wrote:

>Hey
>
>The mission statement is what we've been using for a while. The
>"official title" is new.
>
>  Official Title: OpenStack Common Libraries
>  PTL: Mark McLoughlin 
>  Mission Statement:
>To produce a set of python libraries containing infrastructure code 
>shared by OpenStack projects. The APIs provided by these libraries
>should be high quality, stable, consistent and generally applicable.
>
>I did consider explicitly mentioning technical debt with something like:
>
>  Mission Statement:
>To tackle copy-and-paste technical debt in OpenStack by producing a
>set of python libraries containing infrastructure code shared by
>OpenStack projects. The APIs provided by these libraries should be
>high quality, stable, consistent and generally applicable.
>
>But for wholly new code, that sounds like you need to introduce
>copy-and-paste technical debt before it can be considered in scope for
>Oslo :)
>
>Cheers,
>Mark.
>
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Swift debugging / performance - large latencies seen.

I have 2 openstack clusters running the Folsom release with multiple Swift 
nodes. I also have a small setup that is running only Swift with a single node. 
 I'm noticing very large Swift I/O latencies (seconds long) on the openstack 
clusters - ssbench output snippet is below. Performance is approximately 
identical on the openstack clusters. The Swift only cluster performs much 
better.
Setup differences:
Openstack clusters are using Keystone authentication - Swift only setup uses 
temp auth.
Multiple Swift nodes on openstack clusters - Swift only has single node.

I also see that all object I/Os are being sent to the 2nd Swift node, there are 
no objects on the 1st swift node. Both nodes are running swift-proxy services.
I have LOG_LOCAL0 set in the environment and also logging enabled in 
/etc/swift/swift.conf, but haven't seen any log entries made.

What can I look at to debug the cause of the excessive latencies on both of 
these stacks?
I'd also like to determine why all objects are on the 2nd swift node and none 
on the 1st node.

I'm supposed to do performance evaluations but need to fix the latency problem 
first.

Ssbench output snippet:
TOTAL
   Count:50  Average requests per second:   9.2
min   max  avg  std_dev  95%-ile
   Worst latency TX ID
   First-byte latency:  0.067 -   2.5130.390  (  0.604)1.948  (all 
obj sizes)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.067 -   3.3370.430  (  0.695)1.997  (all 
obj sizes)  txdcedb82227654b338daa85751f6d1232
   First-byte latency:  0.070 -   2.5130.542  (  0.749)2.255  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   Last-byte  latency:  0.070 -   2.5140.468  (  0.659)1.997  (
tiny objs)  txae75691d37d544b4ac0cfe3b8cba7f38
   First-byte latency:  0.067 -   1.8840.251  (  0.382)0.695  (   
small objs)  tx2ceec827f3304530b01a0d5993eea2e8
   Last-byte  latency:  0.067 -   3.3370.385  (  0.732)1.884  (   
small objs)  txdcedb82227654b338daa85751f6d1232

Thanks,
Tim

Timothy Snider
Strategic Planning & Architecture - Advanced Development
NetApp
316-636-8736 Direct Phone
316-213-0223 Mobile Phone
tim.sni...@netapp.com
netapp.com
 [Description: http://media.netapp.com/images/netapp-logo-sig-5.gif]


<>___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Horizon] Navigation UX Enhancements - Collecting Issues

2013-07-09 Thread Jaromir Coufal


Hi everybody,

in UX community group on G+ popped out a need for enhancing user 
experience of main navigation, because there are spreading out various 
issues .


There is already created a BP for this: 
https://blueprints.launchpad.net/horizon/+spec/navigation-enhancement


Toshi had great idea to start discussion about navigation issues on 
mailing list.


So I'd like to ask all of you, if you have some issues with navigation, 
what are the issues you are dealing with? I'd like to gather as much 
feedback as possible, so we can design the best solution which covers 
most of the cases. Issues will be listed in BP and I will try to come 
out with design proposals which hopefully will help all of you.


Examples are following:
* Navigation is not scaling for more dashboards (Project, Admin, ...)
* Each dashboard might contain different hierarchy (number of levels)

What problems do you experience with navigation?

Thanks all for contributing
-- Jarda
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Work around DB in OpenStack (Oslo, Nova, Cinder, Glance)

2013-07-09 Thread Boris Pavlovic

Hi Mark, Nikola, David

Our work is not just in case of unifying. It improves the situation in all
project (not only in Nova).

I would like to say my opinion about DB Archiving also ;)

Let start from the problem, abstract solution, current solution, and why
this solution is ok.

*) Problem.
Records from DB are not deleted at all, so our DB will die.
*) Abstract solution
We should somehow remove old records, I see only one solution, create
shadow tables and have a utilities that are smart and could remove data in
such way, that shadow and main table are absolutly independent.
*) Current solution
1) Create shadow tables
2) Simple utils that move from table to shadow table deleted records

*) Problems in current solution.
If we just move deleted records to shadow table we have to do all joins
(like in Nikola's migration).

So the problem is not in approach of shadow tables, problem is in current
utils that are not enough smart.
And in oslo there is only code (that allows to create_shadow table and that
check that shadow tables and main are synced)

And one more nit such migrations (as made Nikola) are pretty rare.

So I don't see any reason to block this DB Archiving code in oslo and block
this approach. It could be improved not replaced.
More than we are ready to improve it.

Best regards,
Boris Pavlovic

On Tue, Jul 9, 2013 at 3:05 PM, Mark McLoughlin  wrote:

> On Mon, 2013-07-08 at 14:15 +0200, Nikola Đipanov wrote:
> > On 05/07/13 14:26, Boris Pavlovic wrote:
> > > Hi all,
> > >
> > > I would like to explain very high level steps of our work:
> > > 1) Sync work with DB in all projects (We have what we have, let it be
> in
> > > one place)
> > > 2) Refactor work with DB in one place (not independently in all
> projects)
> > >
> > > So I understand that our code around DB is not ideal, but let it be in
> > > one place at first.
> > >
> >
> > This is fine in principle, however I don't think we should push it
> > without considering the details (where the devil is apparently).
> > I am arguing that DB archiving should be re-done and is broken
> > conceptually (example below), and I think it would be suboptimal (to say
> > the least) to get it everywhere first and then fix it.
> >
> > Just saying a hand-wavy "yeah, but once it's in Oslo we can fix it" is
> > wrong - especially for functionality that is younger than the time it
> > will likely take it to 'graduate' Oslo.
>
> I'm not following this DB archiving debate closely enough to take a
> position either way, but 
>
> I think what you're really arguing is that no other project should adopt
> this approach to DB archiving. I'm fine with saying that it shouldn't
> move into oslo-incubator if it will only be used in Nova.
>
> So - the debate to have is which projects are proposing to adopt this DB
> archiving strategy and whether it makes sense for them to adopt it as is
> and fix it up later, or adopt an entirely different approach.
>
> Cheers,
> Mark.
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo

Michael Still wrote:
> On Tue, Jul 9, 2013 at 8:11 PM, Mark McLoughlin  wrote:
>> The mission statement is what we've been using for a while. The
>> "official title" is new.
>>
>>   Official Title: OpenStack Common Libraries
>>   PTL: Mark McLoughlin 
>>   Mission Statement:
>> To produce a set of python libraries containing infrastructure code
>> shared by OpenStack projects. The APIs provided by these libraries
>> should be high quality, stable, consistent and generally applicable.
> 
> This description looks good to me.

+1

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Review request: Blurprint of API validation

2013-07-09 Thread Ken'ichi Ohmichi

Hi,

The blueprint "nova-api-validation-fw" has not been approved yet.
I hope the core patch of this blueprint is merged to Havana-2,
because of completing comprehensive API validation of Nova v3 API
for Havana release. What should we do for it?

The summary of "nova-api-validation-fw":
  * The patch review is in good progress.
  * "nova-api-validation-fw" is API input validation framework.
  * Simplify the code in the extensions of Nova v3 API.
  * Define API schema with JSON Schema.
  * Define common data formats to standardize whole of Nova v3 API.
  * Possible to choose APIs which are applied with this framework.
  * Possible to migrate other web framework.

The details of "nova-api-validation-fw" are the following:

* The patch review is in good progress.
  7 reviewers have marked +1 now.
  We are waiting for the approval of this blueprint before merging the
  patch. (https://review.openstack.org/#/c/25358/)

* "nova-api-validation-fw" is API input validation framework.
  This framework validates API parameters in a request body before the
  execution of API method. If the parameter is invalid, nova-api returns
  a BadRequest response including its reason.  

* Simplify the code in the extensions of Nova v3 API.
  There are a lot of validation code in each API method.
  After applying this framework, we will be able to separate them from
  API methods by defining API schema. That makes the code more readable.
  Also through trying to define API schema of each API, we will find the
  lacks of validation code and complement them for better v3 API.
  Necessary API schemas are for API which contains a request body. They
  are 37 APIs on Nova v3 API now, and they will increase later because
  the tasks, which port v2 API to v3 API, is in progress.

* Define API schema with JSON Schema.
  JSON Schema contains many features for validation, the details are
  written on http://json-schema.org/.
  Here is the schema of v3 keypairs API as a sample:
  == Request body sample of keypairs API ===
  {
"keypair": {
"name": "keypair-dab428fe-6186-4a14-b3de-92131f76cd39",
"public_key": "ssh-rsa B3NzaC1yc2EA[..]== Generated by Nova"
  }
  == API schema of keypairs API 
  {
  'type': 'object',
  'properties': {
  'keypair': {
  'type': 'object',
  'properties': {
  'name': {'type': 'string', 'minLength': 1, 'maxLength': 255},
  'public_key': {'type': 'string'},
  },
  'required': ['name'],
  },
  },
  'required': ['keypair'],
  }

* Define common data formats to standardize whole of Nova v3 API.
  We can define common data formats with FormatChecker of JSON Schema
  library.

  e.g. Data format of 'boolean':
@jsonschema.FormatChecker.cls_checks('boolean')
def validate_boolean_format(instance):
return instance.upper() == 'TRUE' or instance.upper() == 'FALSE'

  The formats can be used in API schema:
'onSharedStorage': {
'type': 'string', 'format': 'boolean',
},

  By re-using these common formats in many API schemas, that will be
  able to standardize whole of Nova v3 API.

* Possible to choose APIs which are applied with this framework.
  We can apply this framework to Nova v3 API only, the existing API (v2) can
  be out of scope of this framework.

* Possible to migrate other web framework.
  The API validation of this framework is executed with the decorator of API
  method, that means this framework does not depend on web framework.
  Also when migrating other web framework(e.g. Pecan/WSME) in the future, we
  will be able to use this framework.

Thanks
Ken'ichi Ohmichi

---
On Fri, 5 Jul 2013 16:27:39 +0900
"Ken'ichi Ohmichi"  wrote:
> 
> Hi,
> 
> I have submitted a blueprint[1] and a patch[2] for Nova API validation.
> I request someone to review the blueprint and hopefully approve it.
> 
> This blueprint was approved once, but the Definition has been changed to
> Discussion because we discussed what is better validation mechanism "WSME
> original validation or JSON Schema". IIUC, we have found the advantages
> of JSON Schema through the discussion. [3]
> 
> Also we have found the way to overlap JSON Schema with WSME, so we will
> be able to use JSON Schema also when the web framework of Nova is changed
> to WSME. [4]
> 
> My latest patch has been implemented with JSON Schema[2], and I think API
> validation with JSON Schema is useful for Nova v3 API, because the API has
> complex API parameters and JSON Schema can check complex parameters 
> naturally. 
> 
> 
> Any comments are welcome :-)
> 
> [1]: 
> https://blueprints.launchpad.net/openstack/?searchtext=nova-api-validation-fw
> [2]: https://review.openstack.org/#/c/25358/
> [3]: http://lists.openstack.org/pipermail/openstack-dev/2013-June/009824.html
>  - http://lists.openstack.org/pipermail/openstack-dev/2013-June/009926.html
>  - http://lists.openstack.org/

Re: [openstack-dev] Update on monitoring vs instrumentation?

2013-07-09 Thread Sandy Walsh

While StackTach is still under active development and has undergone many 
awesome enhancements since that page was written, we're working on porting it 
all to Ceilometer (CM). 

Recently, CM added a UDP Collector, but don't think it's been tested against 
Tach. Tach + statsd + graphite is still a great quick way to get metrics from 
OpenStack, but we hope that CM will be able to do all of that and more. 

Hope it helps.
-S

From: John Wood [john.w...@rackspace.com]
Sent: Tuesday, July 09, 2013 2:42 AM
To: OpenStack Development Mailing List
Subject: [openstack-dev]  Update on monitoring vs instrumentation?

Hello folks,

I've found this page to be informative in trying to understanding monitoring vs 
instrumentation within OpenStack: 
https://wiki.openstack.org/wiki/UnifiedInstrumentationMetering

It is a bit dated though (from October last year I believe). I was curious if 
it still reflects current trends within OpenStack, especially in regards to 
favoring the use of Ceilometer for monitoring/auditing collection services, and 
the use of Tach/Stacktach feeding into Statsd/Graphite for lower level metrics 
gathering and display?

Thanks,
John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] HACKING and new-style relative imports

2013-07-09 Thread Sean Dague


On 07/09/2013 07:28 AM, Kieran Spear wrote:

Hi all,

There's a review up to make Horizon pass H304 (no relative imports):

https://review.openstack.org/#/c/35664/

Old-style relative imports are dangerous, but I can't think of a good enough 
reason to forbid new-style imports, e.g.:

from .views import IndexView

instead of

from openstack_dashboard.dashboards.project.images_and_snapshots.views \
   import IndexView


However, if you decide to address H302, most of the over 80 character 
issues in horizon are going to fit, as this would become:


from openstack_dashboard.dashboards.project.images_and_snapshots import 
views


(tacking in at 78 chars)

I'm sure there will still be a few which break the limit... but that 
will fix a lot of them (from what I see in the review).


-Sean

--
Sean Dague
http://dague.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo

2013-07-09 Thread Michael Still

On Tue, Jul 9, 2013 at 8:11 PM, Mark McLoughlin  wrote:
> Hey
>
> The mission statement is what we've been using for a while. The
> "official title" is new.
>
>   Official Title: OpenStack Common Libraries
>   PTL: Mark McLoughlin 
>   Mission Statement:
> To produce a set of python libraries containing infrastructure code
> shared by OpenStack projects. The APIs provided by these libraries
> should be high quality, stable, consistent and generally applicable.

This description looks good to me.

Michael

--
Rackspace Australia

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] HACKING and new-style relative imports

Hi all,

There's a review up to make Horizon pass H304 (no relative imports):

https://review.openstack.org/#/c/35664/

Old-style relative imports are dangerous, but I can't think of a good enough 
reason to forbid new-style imports, e.g.:

from .views import IndexView

instead of

from openstack_dashboard.dashboards.project.images_and_snapshots.views \
  import IndexView

particularly in Horizon where nesting is deep and you otherwise end up with 
many imports that won't fit into 80 characters. I think if we're going to make 
the change above the benefits would need to outweigh the effect on readability.

There's some discussion on the review already, but I'd like to hear from a more 
general audience. Thoughts?

Cheers,
Kieran


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Work around DB in OpenStack (Oslo, Nova, Cinder, Glance)

On Mon, 2013-07-08 at 14:15 +0200, Nikola Đipanov wrote:
> On 05/07/13 14:26, Boris Pavlovic wrote:
> > Hi all, 
> > 
> > I would like to explain very high level steps of our work: 
> > 1) Sync work with DB in all projects (We have what we have, let it be in
> > one place)
> > 2) Refactor work with DB in one place (not independently in all projects) 
> > 
> > So I understand that our code around DB is not ideal, but let it be in
> > one place at first.
> > 
> 
> This is fine in principle, however I don't think we should push it
> without considering the details (where the devil is apparently).
> I am arguing that DB archiving should be re-done and is broken
> conceptually (example below), and I think it would be suboptimal (to say
> the least) to get it everywhere first and then fix it.
> 
> Just saying a hand-wavy "yeah, but once it's in Oslo we can fix it" is
> wrong - especially for functionality that is younger than the time it
> will likely take it to 'graduate' Oslo.

I'm not following this DB archiving debate closely enough to take a
position either way, but 

I think what you're really arguing is that no other project should adopt
this approach to DB archiving. I'm fine with saying that it shouldn't
move into oslo-incubator if it will only be used in Nova.

So - the debate to have is which projects are proposing to adopt this DB
archiving strategy and whether it makes sense for them to adopt it as is
and fix it up later, or adopt an entirely different approach.

Cheers,
Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Program description for Oslo