Re: [openstack-dev] live-snapshot/cloning of virtual machines

[Bob Ball]

I agree with the below from a XenServer perspective.  As with vmware, XenServer 
supports live snapshotting and creating multiple clones from that live snapshot.

I understand that there is a XenAPI equivalent in the works and therefore would 
argue the API changes need to be accepted as a minimum.

In order to minimize the feature divergence between hypervisors, I'd also argue 
that we should accept the libvirt implementation even if it uses unsupported 
APIs - perhaps disabled by default with a suitable warning that it isn't 
considered safe by libvirt/QEmu.

Bob

From: Shawn Hartsock [hartso...@vmware.com]
Sent: 19 August 2013 20:59
To: Daniel P. Berrange; OpenStack Development Mailing List
Subject: Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

For what it's worth... this doesn't seem too bad to me...

I was planning on using this part of the vSphere API:
* 
https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.CloneSpec.html

...to accomplish the clone part of the BP. The API contains a "spec" section 
where you tell the ESX hypervisor how to handle things like network identity...
* 
https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.customization.IPSettings.html

... I was going to use this to plumb together the "live-snapshot" bit ...
* 
https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.VirtualMachine.html#createSnapshot

... which includes stuff about how to handle memory.

So, I didn't read this blueprint as especially hard to accomplish in the 
vmwareapi driver. It just would eat more time than I have right now and would 
require a deeper level of understanding of how the vSphere hypervisor suite 
works than most of the other features currently use. I'm fully planning on 
playing with this in IceHouse just to see how it would go, it's probably one of 
the more nifty new features we could add.

Note: these are old features for the API and they are a tad complicated, but 
it's all well within the realm of supported! In fact, it's standard operating 
procedure to use the clone feature to scale-out an application in some vSphere 
shops. (albeit, in production the admins I know personally, use clone with 
power-off from a 'template' and the system comes up with a new MAC/etc on first 
boot... cloning from a running system is possible, but I would recommend 
cloning from a power-off state unless you've got a hot-plug feature in your 
guest OS)



# Shawn Hartsock

- Original Message -
> From: "Daniel P. Berrange" 
> To: "OpenStack Development Mailing List" 
> Sent: Monday, August 19, 2013 5:24:59 AM
> Subject: Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines
>
> On Mon, Aug 19, 2013 at 08:28:58AM +1200, Robert Collins wrote:
> > On 17 August 2013 07:01, Russell Bryant  wrote:
> >
> > >> Maybe we've grown up to the point where we have to be more careful and
> > >> not introduce
> > >> these kind of features and the maintenance cost of introducing
> > >> experimental features is
> > >> too great. If that is the community consensus, then I'm happy keep the
> > >> live snapshot stuff
> > >> in a branch on github for people to experiment with.
> > >
> > > My feeling after following this discussion is that it's probably best to
> > > keep baking in another branch (github or whatever).  The biggest reason
> > > is because of the last comment quoted from Daniel Berrange above.  I
> > > feel that like that is a pretty big deal.
> >
> > So, reading between the lines here, I guess you're worried that we'd
> > let code paths that violate what upstream will support leak into the
> > main codepaths for libvirt - and thus we'd end up with a situation
> > where we aren't supported by upstream for all regular operations.
>
> Yes, if you perform a live clone of a VM, then you have effectively
> tainted that VM for the rest of its lifetime. From the virt host
> vendors' POV, any unexpected or problematic behaviour you get from
> that VM thereafter will be outside scope of support. The onus would
> be on the openstack sysadmin to demonstrate that the same problem
> occurs without the use of live cloning.
>
> Running a production cloud using a feature that your virt host
> vendor considers unsupported would be somewhat reckless IMHO, unless
> you think your sysadmins think they have the skills to solve all
> possible problems in that area themselves, which is unlikely for most
> cloud vendors.
>
> Regards,
> Daniel
> --
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org  -o- http://virt-manager.org :|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack

[openstack-dev] [Openstack][keystone][LDAP] Error about "enable" and "desc" attribute type undefine.

[Qinglong.Meng]

Hi all,
  I configure keystone with ldap backend followed LDAP section of
http://docs.openstack.org/developer/keystone/configuration.html,
 and when I create tenant in ldap, I got the error about "enable" and
"desc" attribute type undefined in keystone.log.
 Here is keystone.conf:
 http://paste.openstack.org/show/44574/

 keystone.log
 http://paste.openstack.org/show/44575/

 the ldif of ldap server
 http://paste.openstack.org/show/44578/

 sample slapd.conf
 http://paste.openstack.org/show/44579/

-- 

Lawrency Meng
mail: mengql112...@gmail.com
___
Mailing list: https://launchpad.net/~openstack
Post to : openst...@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Ben Nemec]

 

On 08/19/13 20:34, Joshua Harlow wrote: 

> Just a related question, 
> 
> Oslo 'incubator' db code I think depends on eventlet. This means any code 
> that uses the oslo.db code could/would(?) be dependent on eventlet. 
> 
> Will there be some refactoring there to not require it (useful for projects 
> that are trying to move away from eventlet). 
> 
> https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/session.py#L248
>  [1]

Glancing through that file, it looks like the greenthread import is only
used for playing nice with other greenthreads. It should be pretty easy
to make it conditional so we don't require it, but will use it if it's
available.

 -Ben 

Links:
--
[1]
https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/session.py#L248___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [nova][vmware] VMwareAPI sub-team status update 2013-08-19

[Shawn Hartsock]

Greetings stackers!

August 22nd is fast approaching. Here's the reviews in flight. We have 5 ready 
for a core reviewer to take a look. One needing some attention from someone who 
knows VMware's APIs and 8 that are in need of work/discussion. I noticed that 
there was some issues with Jenkins earlier, some of you may have been caught in 
that. Don't use 'recheck no bug' until you've read the failures and made sure 
it has nothing to do with your patch.

Ready for core reviewer:
* NEW, https://review.openstack.org/#/c/37819/ ,'VMware image clone strategy 
settings and overrides'
https://blueprints.launchpad.net/nova/+spec/vmware-image-clone-strategy
core votes,0, non-core votes,4, down votes, 0
* NEW, https://review.openstack.org/#/c/33100/ ,'Fixes host stats for 
VMWareVCDriver'
https://bugs.launchpad.net/nova/+bug/1190515
core votes,0, non-core votes,5, down votes, 0
* NEW, https://review.openstack.org/#/c/30628/ ,'Fix VCDriver to pick the 
datastore that has capacity'
https://bugs.launchpad.net/nova/+bug/1171930
core votes,0, non-core votes,7, down votes, 0
* NEW, https://review.openstack.org/#/c/33504/ ,'VMware: nova-compute crashes 
if VC not available'
https://bugs.launchpad.net/nova/+bug/1192016
core votes,0, non-core votes,5, down votes, 0
* NEW, https://review.openstack.org/#/c/40298/ ,'Fix snapshot in 
VMWwareVCDriver'
https://bugs.launchpad.net/nova/+bug/1184807
core votes,0, non-core votes,6, down votes, 0

Needs VMware API expert review:
* NEW, https://review.openstack.org/#/c/42024/ ,'VMWare: Disabling linked clone 
doesn't cache images'
https://bugs.launchpad.net/nova/+bug/1207064
core votes,0, non-core votes,0, down votes, 0

Needs discussion/work (has -1):
* NEW, https://review.openstack.org/#/c/37659/ ,'Enhance VMware Hyper instance 
disk usage'
https://blueprints.launchpad.net/nova/+spec/improve-vmware-disk-usage
core votes,0, non-core votes,2, down votes, -1
* NEW, https://review.openstack.org/#/c/39720/ ,'VMware: Added check for 
datastore state before selection'
https://bugs.launchpad.net/nova/+bug/1194078
core votes,0, non-core votes,4, down votes, -1
* NEW, https://review.openstack.org/#/c/40105/ ,'VMware: use VM uuid for volume 
attach and detach'
https://bugs.launchpad.net/nova/+bug/1208173
core votes,1, non-core votes,7, down votes, -1
* NEW, https://review.openstack.org/#/c/41387/ ,'VMware: Nova boot from cinder 
volume'
https://blueprints.launchpad.net/nova/+spec/vmware-nova-cinder-support
core votes,0, non-core votes,2, down votes, -1
* NEW, https://review.openstack.org/#/c/40245/ ,'Nova support for vmware cinder 
driver'
https://blueprints.launchpad.net/nova/+spec/vmware-nova-cinder-support
core votes,0, non-core votes,2, down votes, -1
* NEW, https://review.openstack.org/#/c/41657/ ,'Fix VMwareVCDriver to support 
multi-datastore'
https://bugs.launchpad.net/nova/+bug/1104994
core votes,0, non-core votes,0, down votes, -1
* NEW, https://review.openstack.org/#/c/30282/ ,'Multiple Clusters using single 
compute service'

https://blueprints.launchpad.net/nova/+spec/multiple-clusters-managed-by-one-service
core votes,0, non-core votes,2, down votes, -2
* NEW, https://review.openstack.org/#/c/34903/ ,'Deploy vCenter templates'

https://blueprints.launchpad.net/nova/+spec/deploy-vcenter-templates-from-vmware-nova-driver
core votes,0, non-core votes,2, down votes, -2

Meeting info:
* https://wiki.openstack.org/wiki/Meetings/VMwareAPI


# Shawn Hartsock

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] v3 api remove security_groups extension (was Re: security_groups extension in nova api v3)

[Alex Xu]

On 2013年08月17日 00:14, Vishvananda Ishaya wrote:

On Aug 15, 2013, at 5:58 PM, Melanie Witt  wrote:


On Aug 15, 2013, at 1:13 PM, Joe Gordon wrote:


+1 from me as long as this wouldn't change anything for the EC2 API's security 
groups support, which I assume it won't.

Correct, it's unrelated to the ec2 api.

We discussed briefly in the nova meeting today and there was consensus that 
removing the standalone associate/disassociate actions should happen.

Now the question is whether to keep the server create piece and not remove the 
extension entirely. The concern is about a delay in the newly provisioned 
instance being associated with the desired security groups. With the extension, 
the instance gets the desired security groups before the instance is active (I 
think). Without the extension, the client would receive the active instance and 
then call neutron to associate it with the desired security groups.

Would such a delay in associating with security groups be a problem?


It seems like getting around this would be as simple as:

a. Create the port in neutron.
b. Associate a security group with the port.
c. Boot the instance with the port.

In general I'm a fan of doing all of the network creation and volume creation 
in neutron and cinder before booting the instance. Unfortunately I think this 
is pretty unfriendly to our users. One possibility is to move the smarts into 
the client side (i.e. have it talk to neutron and cinder), but I think that 
alienates all of the people using openstack who are not using python-novaclient 
or python-openstack client.
The API user is developer too, it shouldn't too difficult for them. I 
prefer move the smarts into client
side. I'm open with two way. And I will comment in my patch for notice 
reviewer vote for their flavor way before review my patch.


Since we are still supporting v2 this is a possibility for the v3 api, but if 
you can't do basic operations in v3 without talking to multiple services on the 
client side I think it will prevent a lot of people from using it.

Its clear to me that autocreation needs to stick around for a while just to 
keep the apis usable. I can see the argument for pulling it from the v3 api, 
but it seems like at least the basics need to stick around for now.

Vish


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Code review study

[Michael Davies]

On Tue, Aug 20, 2013 at 5:14 AM, Jay Buffington  wrote:
>
> This is really interesting.  I wish they would have explicitly defined
> "lines of code."   Is that "git show |wc -l"? Just the new lines which
> were added?  The sum of the lines changed, removed and added?  You can
> get vastly different numbers depending on how you count it.

Normally "in the literature" LOC is defined as "non-comment, non-blank
code line deltas, with a few exceptions".

The exceptions normally refer to not counting braces in C-style
languages and other syntactic sugar elements.  Of course in Python we
don't really have these issues to content with :)

I'd normally include comments and docstrings too, since we review these as well.

Michael...
-- 
Michael Davies   mich...@the-davies.net

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Joshua Harlow]

Just a related question,

Oslo 'incubator' db code I think depends on eventlet. This means any code that 
uses the oslo.db code could/would(?) be dependent on eventlet.

Will there be some refactoring there to not require it (useful for projects 
that are trying to move away from eventlet).

https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/session.py#L248

From: Boris Pavlovic mailto:bo...@pavlovic.me>>
Reply-To: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, August 19, 2013 2:12 PM
To: OpenStack Development Mailing List 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

Mark,

But for a variety of reasons, I do not consider the general thrust of "use oslo 
db code" to be approved. Instead, lets continue to consider features from olso 
db on a case by case basis, and see what the right resolution is in each case.

Absolutely agree with this point (e.g. we removed shadow tables from our 
roadmap after some discussion in other threads)
So we are planing to make all changes using our common approach called "baby 
steps" (Not by one giant patch set).

Btw I answered on your question about changed conf parameter in review (I mean 
sql_connection to database.connection).


Best regards,
Boris Pavlovic
---
Mirantis Inc.



On Mon, Aug 19, 2013 at 9:33 PM, Mark Washenberger 
mailto:mark.washenber...@markwash.net>> wrote:
Thanks for refocusing the discussion on your original questions!

Also thanks for this additional summary. I consider the patches you have up for 
review in glance to have a general direction-level green light at this point 
(though I've got a question on the specifics in the ultimate review).

But for a variety of reasons, I do not consider the general thrust of "use oslo 
db code" to be approved. Instead, lets continue to consider features from olso 
db on a case by case basis, and see what the right resolution is in each case.

Thanks for your patience and forbearance, hopefully getting in the patches you 
have submitted now will help unblock progress for your team.

On Mon, Aug 19, 2013 at 3:49 AM, Boris Pavlovic 
mailto:bo...@pavlovic.me>> wrote:
Mark,

Main part of oslo is:
1) common migration testing
2) common sqla.models
3) common hacks around sqla and sqla-migrate
4) common work around engines and sessions


All these points are implemented in Glance almost in the same way as in Oslo.
Also we are able to use only part of this code in Glance, and add some other 
things that are glance related over this code.

Our current 2 patches on review do next things:
1) Copy paste oslo.db code into glance
2) Use sqla session/engine/exception wrappers
3) Remove Glance code that covers session/engine/exception

So I really don't see any bad thing in this code:
1) If you would like to implement other backends => this change won't block it
2) If you would like to make some other sqla utitlites or glance related things 
=> this change won't block it
3) If there are bugs => fix it in oslo and sync => this change won't block it

 So I really don't see any reason to block work around migration to oslo.db 
code in Glance.


Best regards,
Boris Pavlovic
---
Mirantis Inc.




On Fri, Aug 16, 2013 at 10:41 PM, Mark Washenberger 
mailto:mark.washenber...@markwash.net>> wrote:
I would prefer to pick and choose which parts of oslo common db code to reuse 
in glance. Most parts there look great and very useful. However, some parts 
seem like they would conflict with several goals we have.

1) To improve code sanity, we need to break away from the idea of having one 
giant db api interface
2) We need to improve our position with respect to new, non SQL drivers
- mostly, we need to focus first on removing business logic (especially 
authz) from database driver code
- we also need to break away from the strict functional interface, because 
it limits our ability to express query filters and tends to lump all filter 
handling for a given function into a single code block (which ends up being 
defect-rich and confusing as hell to reimplement)
3) It is unfortunate, but I must admit that Glance's code in general is pretty 
heavily coupled to the database code and in particular the schema. Basically 
the only tool we have to manage that problem until we can fix it is to try to 
be as careful as possible about how we change the db code and schema. By 
importing another project, we lose some of that control. Also, even with the 
copy-paste model for oslo incubator, code in oslo does have some of its own 
reasons to change, so we could potentially end up in a conflict where glance db 
migrations (which are operationally costly) have to happen for reasons that 
don't really matter to glance.

So rather than framing this as "glance needs to use oslo common db code", I 
would appreciate framing it as "glance database code should have features X, Y, 
and Z, some of wh

Re: [openstack-dev] Scheduler sub-group meeting on 8/20 - cancelled

[Dugger, Donald D]

Well, no one has stepped up to run the meeting tomorrow so let's just cancel 
for this week.  We'll be back at our regularly scheduled time next Tues., 8/27.

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786


-Original Message-
From: Dugger, Donald D [mailto:donald.d.dug...@intel.com] 
Sent: Friday, August 16, 2013 12:38 PM
To: OpenStack Development Mailing List
Subject: [openstack-dev] Scheduler sub-group meeting on 8/20

Turns out I'll be traveling that day so won't be able to run the meeting.  If 
there's anyone who wants to volunteer to lead the meeting speak now, otherwise 
we can just cancel next week.

--
Don Dugger
"Censeo Toto nos in Kansa esse decisse." - D. Gale
Ph: 303/443-3786



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Interested in a mid-Icehouse-cycle Nova meet-up?

[Shawn Hartsock]

this is off list. Just FYI.

Thanks for putting this up. I was looking forward to going to Hong Kong, but, I 
have a personal (family life event) reason I can't attend. I'll do my best to 
participate otherwise. I've asked Tracy Jones (a vmware team member and ATC) to 
go to Hong Kong in my place.

# Shawn Hartsock

- Original Message -
> From: "Russell Bryant" 
> To: "OpenStack Development Mailing List" 
> Sent: Monday, August 19, 2013 5:42:00 PM
> Subject: [openstack-dev] [Nova] Interested in a mid-Icehouse-cycle Nova   
> meet-up?
> 
> Greetings,
> 
> Some OpenStack programs have started a nice trend of getting together in
> the middle of the development cycle.  These meetups can serve a number
> of useful purposes: community building, ramping up new contributors,
> tackling hard problems by getting together in the same room, and more.
> 
> I am in the early stages of attempting to plan a Nova meet-up for the
> middle of the Icehouse cycle.  To start, I need to get a rough idea of
> how much interest there is.
> 
> I have very little detail at this point, other than I'm looking at
> locations in the US, and that it would be mid-cycle (January/February).
> 
> If you would be interested in this, please fill out the following form:
> 
> http://goo.gl/RPa6iG
> 
> Thanks!
> 
> --
> Russell Bryant
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron] Handling provider removal

[Eugene Nikanorov]

Hi folks,

I'd like to continue the discussion about this.

I think we have the following questions to answer:
1) What should be the workflow of provider removal for the admin?
2) Do we allow 'update' operation on provider attribute?
3) Do we allow removing provider for users?

My take on these:
1) There are two options for the admin. Before restarting neutron-server
with provider removed from conf, they should either:
 - use script to get all resources and filter them by provider. Since
provider technically is a relationship, it can't be done via CLI, so in
fact admin need to filter pools 'manually' (e.g. have some sort of script).
Disassociate each resource from provider.
 - use special API call that will go over all associations with provider
removing them, and doing 'undeploy' operation. I think it's more convenient
way.

Although the patch that has been on review implied the first way of
removing the provider.

2) I think we need to support it.
As simplified form for H-3 we could only allow updates 'no
provider'->'provider'

3) I think we need to support it as well, as there could be various reasons
for users to remove provider.
While there is no provider, resource is handled by 'no-op' plugin driver,
which mere responsibility is to complement db operations of the plugin.
That also means that no-provider resources are fully operable.

Speaking about the patch which is on review, I'm planning to make following
changes:
- implement association of pools and providers within update operation
instead of member action
- implement 'disassociate' admin-only operation, that probably will be some
call on 'providers' collection.

What do you think?

Thanks,
Eugene.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift, netifaces, PyPy, and cffi

[Alex Gaynor]

So, in what can only be described as "extremely embarrassing" and "wow, I
thought I knew how to use a computer": netifaces appears to work ok under
PyPy! I could have sworn I'd tested it, but apparently not. So, this is no
longer a high priority item for me to get swift on pypy (in fact, +/-
 eventlet and pypy releases, the test suite at least all passes!).

I still think long term ditching netifaces is a good idea since it doesn't
really appear to be maintained, but that's a different issue. Since code
churn scares me I'm going to stop really pursuing this, the patch will
always be there if anyone else is excited about this, and maybe someday
I'll get the round tuits to do it myself :)

Alex


On Wed, Aug 14, 2013 at 5:17 PM, Vishvananda Ishaya
wrote:

>
> On Aug 14, 2013, at 11:12 AM, Jarret Raim 
> wrote:
>
>  I vote for including cffi. We are going to use a cffi lib as part of
> Barbican (key management) anyway, so I'd like to see wider acceptance.
>
>  Jarret
>
>
> +1
>
> cffi rocks
>
> Vish
>
>
>   From: Alex Gaynor 
> Reply-To: OpenStack List 
> Date: Wednesday, August 14, 2013 12:12 PM
> To: "openst...@nemebean.com" , OpenStack List <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] Swift, netifaces, PyPy, and cffi
>
>   I just chatted with the Python product owner at Red Hat, he says this
> is going to make it's way to the next step later today (this past weekend
> was a Fedora conference), so this should be happening soon.
>
>  Joe: Yup, I'm familiar with that piece (I had lunch with Vish the other
> week and he's the one who suggested Swift as the best place to get started
> with OpenStack + PyPy). For those who don't know I'm one of the core
> developers of PyPy :)
>
>  Alex
>
>
>
> On Wed, Aug 14, 2013 at 9:24 AM, Ben Nemec  wrote:
>
>> On 2013-08-13 16:58, Alex Gaynor wrote:
>>
>>> One of the issues that came up in this review however, is that cffi is
>>> not packaged in the most recent Ubuntu LTS (and likely other
>>> distributions), although it is available in raring, and in a PPA
>>>  
>>> (http://packages.ubuntu.com/**raring/python-cffi[2]
>>>  nd https://launchpad.net/~**pypy/+archive/ppa?field.**
>>> series_filter=precise
>>> [3] respectively).
>>>
>>>
>>> As a result of this, we wanted to get some feedback on which direction
>>> is best to go:
>>>
>>> a) cffi-only approach, this is obviously the simplest approach, and
>>> works everywhere (assuming you can install a PPA, use pip, or similar
>>> for cffi)
>>> b) wait until the next LTS to move to this approach (requires waiting
>>> until 2014 for PyPy support)
>>> c) Support using either netifaces or cffi: most complex, and most
>>> code, plus "one or the other" dependencies aren't well supported by
>>> most tools as far as I know.
>>>
>>
>> It doesn't appear to me that this is available for RHEL yet, although it
>> looks like they're working on it: https://admin.fedoraproject.**
>> org/updates/python-cffi-0.6-4.**el6
>>
>> That's also going to need to happen before we can do this, I think.
>>
>> -Ben
>>
>>
>> __**_
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.**org 
>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev
>>
>
>
>
>  --
> "I disapprove of what you say, but I will defend to the death your right
> to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: 125F 5C67 DFE9 4084
>___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][VPNaaS] Supporting OpenSwan or StrongSwan or Both?

[Nachi Ueno]

Hi Salvatore

Thank you for your comment.
I'm adding OpenSwan support as additional driver, so it is safe for strongswan.

Best
Nachi

2013/8/19 Salvatore Orlando :
> As I said during the meeting, I am happy to support both as long as the code
> churn is reasonably contained and the chances of strongswan support
> introducing bugs into openswan driver are negligible.
>
> Openswan should be the default solution, in muy opinion.
>
> Salvatore
>
>
> On 20 August 2013 00:15, Nachi Ueno  wrote:
>>
>> Hi folks
>>
>> I would like to discuss whether supporting OpenSwan or StrongSwan or Both
>> for
>> ipsec driver?
>>
>> We choose StrongSwan because of the community is active and plenty of
>> docs.
>> However It looks like RHEL is only supporting OpenSwan.
>>
>> so we should choose
>>
>> (A) Support StrongSwan
>> (B) Support OpenSwan
>> (C) Support both
>>(C-1) Make StrongSwan default
>>(C-2) Make OpenSwan default
>>
>> Actually, I'm working on C-2.
>> The patch is still WIP https://review.openstack.org/#/c/42264/
>>
>> Besides the patch is small, supporting two driver may burden
>> in H3 including docs or additional help.
>> IMO, this is also a valid comment.
>>
>> Best
>> Nachi
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][VPNaaS] Supporting OpenSwan or StrongSwan or Both?

[Salvatore Orlando]

As I said during the meeting, I am happy to support both as long as the
code churn is reasonably contained and the chances of strongswan support
introducing bugs into openswan driver are negligible.

Openswan should be the default solution, in muy opinion.

Salvatore


On 20 August 2013 00:15, Nachi Ueno  wrote:

> Hi folks
>
> I would like to discuss whether supporting OpenSwan or StrongSwan or Both
> for
> ipsec driver?
>
> We choose StrongSwan because of the community is active and plenty of docs.
> However It looks like RHEL is only supporting OpenSwan.
>
> so we should choose
>
> (A) Support StrongSwan
> (B) Support OpenSwan
> (C) Support both
>(C-1) Make StrongSwan default
>(C-2) Make OpenSwan default
>
> Actually, I'm working on C-2.
> The patch is still WIP https://review.openstack.org/#/c/42264/
>
> Besides the patch is small, supporting two driver may burden
> in H3 including docs or additional help.
> IMO, this is also a valid comment.
>
> Best
> Nachi
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][VPNaaS] Supporting OpenSwan or StrongSwan or Both?

[Nachi Ueno]

Hi folks

I would like to discuss whether supporting OpenSwan or StrongSwan or Both for
ipsec driver?

We choose StrongSwan because of the community is active and plenty of docs.
However It looks like RHEL is only supporting OpenSwan.

so we should choose

(A) Support StrongSwan
(B) Support OpenSwan
(C) Support both
   (C-1) Make StrongSwan default
   (C-2) Make OpenSwan default

Actually, I'm working on C-2.
The patch is still WIP https://review.openstack.org/#/c/42264/

Besides the patch is small, supporting two driver may burden
in H3 including docs or additional help.
IMO, this is also a valid comment.

Best
Nachi

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] Interested in a mid-Icehouse-cycle Nova meet-up?

[Russell Bryant]

Greetings,

Some OpenStack programs have started a nice trend of getting together in
the middle of the development cycle.  These meetups can serve a number
of useful purposes: community building, ramping up new contributors,
tackling hard problems by getting together in the same room, and more.

I am in the early stages of attempting to plan a Nova meet-up for the
middle of the Icehouse cycle.  To start, I need to get a rough idea of
how much interest there is.

I have very little detail at this point, other than I'm looking at
locations in the US, and that it would be mid-cycle (January/February).

If you would be interested in this, please fill out the following form:

http://goo.gl/RPa6iG

Thanks!

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Boris Pavlovic]

Mark,

But for a variety of reasons, I do not consider the general thrust of "use
oslo db code" to be approved. Instead, lets continue to consider features
from olso db on a case by case basis, and see what the right resolution is
in each case.

Absolutely agree with this point (e.g. we removed shadow tables from our
roadmap after some discussion in other threads)
So we are planing to make all changes using our common approach called
"baby steps" (Not by one giant patch set).

Btw I answered on your question about changed conf parameter in review (I
mean sql_connection to database.connection).


Best regards,
Boris Pavlovic
---
Mirantis Inc.



On Mon, Aug 19, 2013 at 9:33 PM, Mark Washenberger <
mark.washenber...@markwash.net> wrote:

> Thanks for refocusing the discussion on your original questions!
>
> Also thanks for this additional summary. I consider the patches you have
> up for review in glance to have a general direction-level green light at
> this point (though I've got a question on the specifics in the ultimate
> review).
>
> But for a variety of reasons, I do not consider the general thrust of "use
> oslo db code" to be approved. Instead, lets continue to consider features
> from olso db on a case by case basis, and see what the right resolution is
> in each case.
>
> Thanks for your patience and forbearance, hopefully getting in the patches
> you have submitted now will help unblock progress for your team.
>
> On Mon, Aug 19, 2013 at 3:49 AM, Boris Pavlovic  wrote:
>
>> Mark,
>>
>> Main part of oslo is:
>> 1) common migration testing
>> 2) common sqla.models
>> 3) common hacks around sqla and sqla-migrate
>> 4) common work around engines and sessions
>>
>>
>> All these points are implemented in Glance almost in the same way as in
>> Oslo.
>> Also we are able to use only part of this code in Glance, and add some
>> other things that are glance related over this code.
>>
>> Our current 2 patches on review do next things:
>> 1) Copy paste oslo.db code into glance
>> 2) Use sqla session/engine/exception wrappers
>> 3) Remove Glance code that covers session/engine/exception
>>
>> So I really don't see any bad thing in this code:
>> 1) If you would like to implement other backends => this change won't
>> block it
>> 2) If you would like to make some other sqla utitlites or glance related
>> things => this change won't block it
>> 3) If there are bugs => fix it in oslo and sync => this change won't
>> block it
>>
>>  So I really don't see any reason to block work around migration to
>> oslo.db code in Glance.
>>
>>
>> Best regards,
>> Boris Pavlovic
>> ---
>> Mirantis Inc.
>>
>>
>>
>>
>> On Fri, Aug 16, 2013 at 10:41 PM, Mark Washenberger <
>> mark.washenber...@markwash.net> wrote:
>>
>>> I would prefer to pick and choose which parts of oslo common db code to
>>> reuse in glance. Most parts there look great and very useful. However, some
>>> parts seem like they would conflict with several goals we have.
>>>
>>> 1) To improve code sanity, we need to break away from the idea of having
>>> one giant db api interface
>>> 2) We need to improve our position with respect to new, non SQL drivers
>>> - mostly, we need to focus first on removing business logic
>>> (especially authz) from database driver code
>>> - we also need to break away from the strict functional interface,
>>> because it limits our ability to express query filters and tends to lump
>>> all filter handling for a given function into a single code block (which
>>> ends up being defect-rich and confusing as hell to reimplement)
>>> 3) It is unfortunate, but I must admit that Glance's code in general is
>>> pretty heavily coupled to the database code and in particular the schema.
>>> Basically the only tool we have to manage that problem until we can fix it
>>> is to try to be as careful as possible about how we change the db code and
>>> schema. By importing another project, we lose some of that control. Also,
>>> even with the copy-paste model for oslo incubator, code in oslo does have
>>> some of its own reasons to change, so we could potentially end up in a
>>> conflict where glance db migrations (which are operationally costly) have
>>> to happen for reasons that don't really matter to glance.
>>>
>>> So rather than framing this as "glance needs to use oslo common db
>>> code", I would appreciate framing it as "glance database code should have
>>> features X, Y, and Z, some of which it can get by using oslo code." Indeed,
>>> I believe in IRC we discussed the idea of writing up a wiki listing these
>>> feature improvements, which would allow a finer granularity for evaluation.
>>> I really prefer that format because it feels more like planning and less
>>> like debate :-)
>>>
>>>  I have a few responses inline below.
>>>
>>> On Fri, Aug 16, 2013 at 6:31 AM, Victor Sergeyev >> > wrote:
>>>
 Hello All.

 Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some
 questions about Oslo DB code, and why i

[openstack-dev] [Infra] Meeting Tuesday August 20th at 19:00 UTC

[Elizabeth Krumbach Joseph]

The OpenStack Infrastructure (Infra) team is hosting our weekly
meeting tomorrow, Tuesday August 20th, at 19:00 UTC in
#openstack-meeting

Meeting agenda available here:
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting (anyone is
welcome to to add agenda items)

Everyone interested in infrastructure and process surrounding
automated testing and deployment is encouraged to attend.


-- 
Elizabeth Krumbach Joseph || Lyz || pleia2
http://www.princessleia.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [savanna] tarballs of savanna-extra

[Sergey Lukjanov]

Hi Matt,

it is not an accident that savanna-extra has no tarballs at tarballs.o.o, 
because this repo is used for storing some date that is only needed for some 
stuff like building images for vanilla plugin, storing Swift support patch for 
Hadoop and etc. So, it looks like that we should not package all of them to one 
heterogeneous tarball.

Sincerely yours,
Sergey Lukjanov
Savanna Technical Lead
Mirantis Inc.

On Aug 20, 2013, at 0:25, Matthew Farrellee  wrote:

> Will someone setup a tarballs.os.o release of savanna-extra's master 
> (https://github.com/stackforge/savanna-extra), and make sure it gets an 
> official release for 0.3?
> 
> Best,
> 
> 
> matt
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[John Bresnahan]

> All I'm saying is that we should be careful not to swap one set of
> problems for another. 

My 2 cents: I am in agreement with Jay.  I am leery of NoSQL being a
direct sub in and I fear that this effort can be adding a large workload
for little benefit.

A somewhat related post:
http://www.joelonsoftware.com/articles/fog69.html

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [savanna] tarballs of savanna-extra

[Matthew Farrellee]

Will someone setup a tarballs.os.o release of savanna-extra's master 
(https://github.com/stackforge/savanna-extra), and make sure it gets an 
official release for 0.3?


Best,


matt

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

[Shawn Hartsock]

For what it's worth... this doesn't seem too bad to me...

I was planning on using this part of the vSphere API:
* 
https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.CloneSpec.html

...to accomplish the clone part of the BP. The API contains a "spec" section 
where you tell the ESX hypervisor how to handle things like network identity...
* 
https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.vm.customization.IPSettings.html

... I was going to use this to plumb together the "live-snapshot" bit ...
* 
https://www.vmware.com/support/developer/vc-sdk/visdk400pubs/ReferenceGuide/vim.VirtualMachine.html#createSnapshot

... which includes stuff about how to handle memory.

So, I didn't read this blueprint as especially hard to accomplish in the 
vmwareapi driver. It just would eat more time than I have right now and would 
require a deeper level of understanding of how the vSphere hypervisor suite 
works than most of the other features currently use. I'm fully planning on 
playing with this in IceHouse just to see how it would go, it's probably one of 
the more nifty new features we could add.

Note: these are old features for the API and they are a tad complicated, but 
it's all well within the realm of supported! In fact, it's standard operating 
procedure to use the clone feature to scale-out an application in some vSphere 
shops. (albeit, in production the admins I know personally, use clone with 
power-off from a 'template' and the system comes up with a new MAC/etc on first 
boot... cloning from a running system is possible, but I would recommend 
cloning from a power-off state unless you've got a hot-plug feature in your 
guest OS)



# Shawn Hartsock

- Original Message -
> From: "Daniel P. Berrange" 
> To: "OpenStack Development Mailing List" 
> Sent: Monday, August 19, 2013 5:24:59 AM
> Subject: Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines
> 
> On Mon, Aug 19, 2013 at 08:28:58AM +1200, Robert Collins wrote:
> > On 17 August 2013 07:01, Russell Bryant  wrote:
> > 
> > >> Maybe we've grown up to the point where we have to be more careful and
> > >> not introduce
> > >> these kind of features and the maintenance cost of introducing
> > >> experimental features is
> > >> too great. If that is the community consensus, then I'm happy keep the
> > >> live snapshot stuff
> > >> in a branch on github for people to experiment with.
> > >
> > > My feeling after following this discussion is that it's probably best to
> > > keep baking in another branch (github or whatever).  The biggest reason
> > > is because of the last comment quoted from Daniel Berrange above.  I
> > > feel that like that is a pretty big deal.
> > 
> > So, reading between the lines here, I guess you're worried that we'd
> > let code paths that violate what upstream will support leak into the
> > main codepaths for libvirt - and thus we'd end up with a situation
> > where we aren't supported by upstream for all regular operations.
> 
> Yes, if you perform a live clone of a VM, then you have effectively
> tainted that VM for the rest of its lifetime. From the virt host
> vendors' POV, any unexpected or problematic behaviour you get from
> that VM thereafter will be outside scope of support. The onus would
> be on the openstack sysadmin to demonstrate that the same problem
> occurs without the use of live cloning.
> 
> Running a production cloud using a feature that your virt host
> vendor considers unsupported would be somewhat reckless IMHO, unless
> you think your sysadmins think they have the skills to solve all
> possible problems in that area themselves, which is unlikely for most
> cloud vendors.
> 
> Regards,
> Daniel
> --
> |: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org  -o- http://virt-manager.org :|
> |: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Code review study

[Jay Buffington]

On Wed, Aug 14, 2013 at 7:12 PM, Robert Collins
wrote:

> Note specifically the citation of 200-400 lines as the knee of the review
> effectiveness curve: that's lower than I thought - I thought 200 was
> clearly fine - but no.
>

This is really interesting.  I wish they would have explicitly defined
"lines of code."   Is that "git show |wc -l"? Just the new lines which
were added?  The sum of the lines changed, removed and added?  You can
get vastly different numbers depending on how you count it.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [glance] [ceilometer] Periodic Auditing In Glance

[Sandy Walsh]

On 08/16/2013 04:58 PM, Doug Hellmann wrote:
> The notification messages don't translate 1:1 to database records. Even
> if the notification payload includes multiple resources, we will store
> those as multiple individual records so we can query against them. So it
> seems like sending individual notifications would let us distribute the
> load of processing the notifications across several collector instances,
> and won't have any effect on the data storage requirements.


Well, they would. Each .exists would result in a Event record being
stored with the underlying raw json (pretty big) at the very least.

Like Alex said, if each customer has a daily week-long rolling backup
over 100k instances that's 700k .exists records.

We have some ideas we're kicking around internally about alternative
approaches, but right now I think we have to design for 1 glance .exists
per image (worst case) or 1 glance event per tenant (better case) and
hope that deploying per-cell will help spread the load ... but it'll
suck for making aggregated reports per region.

Phil, like Doug said, I don't think switching from per-instance to
per-tenant or anything else will really affect the end result. The
event->meter mapping will have to break it down anyway.


-S

> 
> Doug
> 
> 
> On Thu, Aug 15, 2013 at 11:58 AM, Alex Meade  > wrote:
> 
> I don't know any actual numbers but I would have the concern that
> images tend to stick around longer than instances. For example, if
> someone takes daily snapshots of their server and keeps them around
> for a long time, the number of exists events would go up and up.
> 
> Just a thought, could be a valid avenue of concern.
> 
> -Alex
> 
> -Original Message-
> From: "Doug Hellmann"  >
> Sent: Thursday, August 15, 2013 11:17am
> To: "OpenStack Development Mailing List"
>  >
> Subject: Re: [openstack-dev] [glance] [ceilometer] Periodic Auditing
> In Glance
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> Nova generates a single exists event for each instance, and that doesn't
> cause a lot of trouble as far as I've been able to see.
> 
> What is the relative number of images compared to instances in a
> "typical"
> cloud?
> 
> Doug
> 
> 
> On Tue, Aug 13, 2013 at 7:20 PM, Neal, Phil  > wrote:
> 
> > I'm a little concerned that a batch payload won't align with "exists"
> > events generated from other services. To my recollection, Cinder,
> Trove and
> > Neutron all emit exists events on a per-instance basisa
> consumer would
> > have to figure out a way to handle/unpack these separately if they
> needed a
> > granular feed. Not the end of the world, I suppose, but a bit
> inconsistent.
> >
> > And a minor quibble: batching would also make it a much bigger
> issue if a
> > consumer missed a notificationthough I guess you could
> counteract that
> > by increasing the frequency (but wouldn't that defeat the purpose?)
> >
> > >
> > >
> > >
> > > On 08/13/2013 04:35 PM, Andrew Melton wrote:
> > > >> I'm just concerned with the type of notification you'd send.
> It has to
> > > >> be enough fine grained so we don't lose too much information.
> > > >
> > > > It's a tough situation, sending out an image.exists for each
> image with
> > > > the same payload as say image.upload would likely create TONS of
> > traffic.
> > > > Personally, I'm thinking about a batch payload, with a bare
> minimum of
> > the
> > > > following values:
> > > >
> > > > 'payload': [{'id': 'uuid1', 'owner': 'tenant1', 'created_at':
> > > > 'some_date', 'size': 1},
> > > >{'id': 'uuid2', 'owner': 'tenant2', 'created_at':
> > > > 'some_date', 'deleted_at': 'some_other_date', 'size': 2}]
> > > >
> > > > That way the audit job/task could be configured to emit in batches
> > which
> > > > a deployer could tweak the settings so as to not emit too many
> > messages.
> > > > I definitely welcome other ideas as well.
> > >
> > > Would it be better to group by tenant vs. image?
> > >
> > > One .exists per tenant that contains all the images owned by
> that tenant?
> > >
> > > -S
> > >
> > >
> > > > Thanks,
> > > > Andrew Melton
> > > >
> > > >
> > > > On Tue, Aug 13, 2013 at 4:27 AM, Julien Danjou
> mailto:jul...@danjou.info>
> > > > >> wrote:
> > > >
> > > > On Mon, 

Re: [openstack-dev] Continuous deployment - significant process change

[Thierry Carrez]

Robert Collins wrote:
> [...]
> But to answer your points:
>  - an unreleased feature is an unreleased feature irrespective of what
> branch it is in. It's not something you'd pull bug fixes into a frozen
> branch for.
>  - If the codebase knows it's a release branch (can we tell
> programmatically?) we could just refuse to enable unreleased features.

The codebase used to be able to tell (Final=True) but with our
tag-driven process it no longer can. That said I don't think it needs
to. A generic use_experimental_features_OMG_DONT_DO_THAT_OH_FOOL flag
could be used in all branches (defaulting to False).

> I understand your concerns about making it easier for one group vs
> another - I don't think this is a zero-sum game. CD disciplines make
> code evolution faster and safer for /everyone/.

I'm totally convinced it's not a zero-sum game. Personally I think we
need a pretty awesome framework to handle experimental configuration
options, one that would make sure you know what you're doing (and the
consequences of doing so) when you enable one. We also need to be able
to tell (from a stable maintenance / vulnerability management
perspective) if code is experimental or not, and manage its lifecycle
easily (transition from experimental to supported or dead).

If that is achieved, the consequences for the user and for the
maintenance teams are limited and probably make it a good trade-off.

> [...]
> The key bad outcomes folk want to see addressed are:
>  - The project owning lots of incomplete useless code
>  - Bugs being introduced
>  - Documentation and user support burdens being increased
>  - Conf file setting proliferation [with backwards compat constraints
> on removals]
>  - Stable branch maintenance becoming harder.

I didn't see where your proposal addresses the "project owning lots of
incomplete useless code" bad outcome. I suspect it would not just be
"incomplete useless code", but also "always experimental code": stuff
that lives in experimental, kinda works, but nobody polishes it or has
the guts to remove it (after all, it kinda works). You have to track
that and solve it somehow.

I suspect this (which I called "experimental feature lifecycle" above)
puts extra tracking effort on PTLs (and ultimately the release manager).
We don't really want extra effort... Ideas ?

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Help consuming trusts

[Steven Hardy]

On Mon, Aug 19, 2013 at 09:15:24AM -0500, Dolph Mathews wrote:
> On Mon, Aug 19, 2013 at 6:06 AM, Steven Hardy  wrote:
> 
> > On Sun, Aug 18, 2013 at 07:02:04PM +0200, Matthieu Huin wrote:
> > > Hi Steve,
> > >
> > > It might be a bit late for this, but here's a script I wrote when
> > experimenting with trusts:
> > https://github.com/mhuin/keystone_trust/blob/master/tests/swift_example.sh
> > >
> > > I hope it'll help you.
> >
> > Thanks for this!!
> >
> > Exactly what I was looking for and has enabled me to solve my problem (my
> > test code was broken).
> >
> > I've marked this bug invalid:
> >
> > https://bugs.launchpad.net/keystone/+bug/1213340
> >
> > Interestingly, my debugging has highlighted a slightly non-obvious issue
> > with
> > the creation and consumption of a trust which is probably worth mentioning
> > here:
> >
> > The docs state ""A project_id may not be specified without at least one
> > role,
> > and vice versa.", however /OS-TRUST/trusts *does* allow you to create a
> > trust
> > with an empty roles list.
> >
> > This results in 401 responses whenever you try to consume the trust, which
> > is
> > not exactly obvious until you realize what's happening..
> >
> > Can I ask if this is deliberate, or is it a bug in the trusts create code?
> >
> 
> That certainly sounds like a bug, given that it directly conflicts with the
> documented behavior.

Ok, thanks, I've raised a bug:

https://bugs.launchpad.net/keystone/+bug/1214064

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat

[Steven Hardy]

I think Zane's response pretty much covers it, but here's some comments
since you requested my response:

On Thu, Aug 15, 2013 at 05:50:19PM -0500, Christopher Armstrong wrote:
> *Introduction and Requirements*
> 
> So there's kind of a perfect storm happening around autoscaling in Heat
> right now. It's making it really hard to figure out how I should compose
> this email. There are a lot of different requirements, a lot of different
> cool ideas, and a lot of projects that want to take advantage of
> autoscaling in one way or another: Trove, OpenShift, TripleO, just to name
> a few...
> 
> I'll try to list the requirements from various people/projects that may be
> relevant to autoscaling or scaling in general.
> 
> 1. Some users want a service like Amazon's Auto Scaling or Rackspace's
> Otter -- a simple API that doesn't really involve orchestration.
> 2. If such a API exists, it makes sense for Heat to take advantage of its
> functionality instead of reimplementing it.
> 3. If Heat integrates with that separate API, however, that API will need
> two ways to do its work:
>1. native instance-launching functionality, for the "simple" use
>2. a way to talk back to Heat to perform orchestration-aware scaling
> operations.
> 4. There may be things that are different than AWS::EC2::Instance that we
> would want to scale (I have personally been playing around with the concept
> of a ResourceGroup, which would maintain a nested stack of resources based
> on an arbitrary template snippet).
> 5. Some people would like to be able to perform manual operations on an
> instance group -- such as Clint Byrum's recent example of "remove instance
> 4 from resource group A".
> 
> Please chime in with your additional requirements if you have any! Trove
> and TripleO people, I'm looking at you :-)
> 
> 
> *TL;DR*
> 
> Point 3.2. above is the main point of this email: exactly how should the
> autoscaling API talk back to Heat to tell it to add more instances? I
> included the other points so that we keep them in mind while considering a
> solution.
> 
> *Possible Solutions*
> 
> I have heard at least three possibilities so far:
> 
> 1. the autoscaling API should maintain a full template of all the nodes in
> the autoscaled nested stack, manipulate it locally when it wants to add or
> remove instances, and post an update-stack to the nested-stack associated
> with the InstanceGroup.
> 
> Pros: It doesn't require any changes to Heat.
> 
> Cons: It puts a lot of burden of state management on the autoscale API, and
> it arguably spreads out the responsibility of "orchestration" to the
> autoscale API. Also arguable is that automated agents outside of Heat
> shouldn't be managing an "internal" template, which are typically developed
> by devops people and kept in version control.

I think this is definitely the way to go - essentially the definition of
the nested stack template now used inside InstanceGroup moves the the
AutoScaling API, and InstanceGroup gets reimplemented in terms of calls to
the AutoScaling API/service.

As Zane illustrated, this allows a clean layered architecture, and
minimises duplication of functionality/code.

I don't think it's any problem that the AS API internally defines a
template, we're just moving the (mechanical) definition of that template
from one place to another, which it seems it the entire aim of this work
(ie decoupling the AutoScaling/InstanceGroup implementation from the
heat-engine)

> 2. There should be a new custom-built API for doing exactly what the
> autoscaling service needs on an InstanceGroup, named something unashamedly
> specific -- like "instance-group-adjust".
> 
> Pros: It'll do exactly what it needs to do for this use case; very little
> state management in autoscale API; it lets Heat do all the orchestration
> and only give very specific delegation to the external autoscale API.
> 
> Cons: The API grows an additional method for a specific use case.
> 
> 3. the autoscaling API should update the "Size" Property of the
> InstanceGroup resource in the stack that it is placed in. This would
> require the ability to PATCH a specific piece of a template (an operation
> isomorphic to update-stack).

As above, I think this is backwards, the AutoScaling API will be a layer
below the InstanceGroup resource, it should definitely not be implemented
using it, or modify the nested stack as you mention above.

Re the comments over RPC/ReST interface - it seems to me, if we're going to
all the effort to decouple this, that the AS API should ideally talk to
Heat via the public ReST inteface.

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Stats on blueprint design info / creation times

[Thierry Carrez]

Daniel P. Berrange wrote:
> In this thread about code review:
> 
>   http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html
> 
> I mentioned that I thought there were too many blueprints created without
> sufficient supporting design information and were being used for "tickbox"
> process compliance only. I based this assertion on a gut feeling I have
> from experiance in reviewing.
> [...]

Nice analysis, Daniel.

One side of this issue is that the blueprints tool no longer matches our
needs (can't have a blueprint that affects multiple projects, can't
discuss in blueprints the same way we do with bugs...).

So I suspect part of the "tickbox" effect is due to people not getting
enough value from blueprints. They are essential for project management
types (think PTLs or me), but feel like a process tickbox for everyone
else. I hope that StoryBoard will one day fix that for us.

> I do think we have scope for being more rigourous in our review of
> blueprints, asking people to expand on the design info associated with
> a blueprint. Perhaps also require that a blueprint is actually approved
> by the core team before we go to the trouble of reviewing & approving
> the code implementing a blueprint in Gerrit.

The approval process has been simplified lately: if a blueprint is
targeted to a milestone and has a priority set (not "Undefined") then it
is considered approved. I agree you could require that the blueprint was
reviewed/prioritized before landing a feature associated with it.

Note that in some cases, some "improvements" that do not clearly fall
into the "bug" category are landed without a blueprint link (or a bug
link). So a first step could be to require that a review always
references a bug or a blueprint before it's landed. Then, improve the
quality of the information present in said bug/blueprint.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Mark Washenberger]

Thanks for refocusing the discussion on your original questions!

Also thanks for this additional summary. I consider the patches you have up
for review in glance to have a general direction-level green light at this
point (though I've got a question on the specifics in the ultimate review).

But for a variety of reasons, I do not consider the general thrust of "use
oslo db code" to be approved. Instead, lets continue to consider features
from olso db on a case by case basis, and see what the right resolution is
in each case.

Thanks for your patience and forbearance, hopefully getting in the patches
you have submitted now will help unblock progress for your team.

On Mon, Aug 19, 2013 at 3:49 AM, Boris Pavlovic  wrote:

> Mark,
>
> Main part of oslo is:
> 1) common migration testing
> 2) common sqla.models
> 3) common hacks around sqla and sqla-migrate
> 4) common work around engines and sessions
>
>
> All these points are implemented in Glance almost in the same way as in
> Oslo.
> Also we are able to use only part of this code in Glance, and add some
> other things that are glance related over this code.
>
> Our current 2 patches on review do next things:
> 1) Copy paste oslo.db code into glance
> 2) Use sqla session/engine/exception wrappers
> 3) Remove Glance code that covers session/engine/exception
>
> So I really don't see any bad thing in this code:
> 1) If you would like to implement other backends => this change won't
> block it
> 2) If you would like to make some other sqla utitlites or glance related
> things => this change won't block it
> 3) If there are bugs => fix it in oslo and sync => this change won't block
> it
>
>  So I really don't see any reason to block work around migration to
> oslo.db code in Glance.
>
>
> Best regards,
> Boris Pavlovic
> ---
> Mirantis Inc.
>
>
>
>
> On Fri, Aug 16, 2013 at 10:41 PM, Mark Washenberger <
> mark.washenber...@markwash.net> wrote:
>
>> I would prefer to pick and choose which parts of oslo common db code to
>> reuse in glance. Most parts there look great and very useful. However, some
>> parts seem like they would conflict with several goals we have.
>>
>> 1) To improve code sanity, we need to break away from the idea of having
>> one giant db api interface
>> 2) We need to improve our position with respect to new, non SQL drivers
>> - mostly, we need to focus first on removing business logic
>> (especially authz) from database driver code
>> - we also need to break away from the strict functional interface,
>> because it limits our ability to express query filters and tends to lump
>> all filter handling for a given function into a single code block (which
>> ends up being defect-rich and confusing as hell to reimplement)
>> 3) It is unfortunate, but I must admit that Glance's code in general is
>> pretty heavily coupled to the database code and in particular the schema.
>> Basically the only tool we have to manage that problem until we can fix it
>> is to try to be as careful as possible about how we change the db code and
>> schema. By importing another project, we lose some of that control. Also,
>> even with the copy-paste model for oslo incubator, code in oslo does have
>> some of its own reasons to change, so we could potentially end up in a
>> conflict where glance db migrations (which are operationally costly) have
>> to happen for reasons that don't really matter to glance.
>>
>> So rather than framing this as "glance needs to use oslo common db code",
>> I would appreciate framing it as "glance database code should have features
>> X, Y, and Z, some of which it can get by using oslo code." Indeed, I
>> believe in IRC we discussed the idea of writing up a wiki listing these
>> feature improvements, which would allow a finer granularity for evaluation.
>> I really prefer that format because it feels more like planning and less
>> like debate :-)
>>
>>  I have a few responses inline below.
>>
>> On Fri, Aug 16, 2013 at 6:31 AM, Victor Sergeyev 
>> wrote:
>>
>>> Hello All.
>>>
>>> Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some
>>> questions about Oslo DB code, and why is it so important to use it instead
>>> of custom implementation and so on. As there were a lot of questions it was
>>> really hard to answer on all this questions in IRC. So we decided that
>>> mailing list is better place for such things.
>>>
>>> List of main questions:
>>>
>>> 1. What includes oslo DB code?
>>> 2. Why is it safe to replace custom implementation by Oslo DB code?
>>> 3. Why oslo DB code is better than custom implementation?
>>> 4. Why oslo DB code won’t slow up project development progress?
>>> 5. What we are going actually to do in Glance?
>>> 6. What is the current status?
>>>
>>> Answers:
>>>
>>> 1. What includes oslo DB code?
>>>
>>> Currently Oslo code improves different aspects around DB:
>>> -- Work with SQLAlchemy models, engine and session
>>> -- Lot of tools for work with SQLAlchemy
>>>
>> -- Work with unique keys
>>>

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Joshua Harlow]

+1 for trying things differently :)

On 8/19/13 12:14 AM, "Jay Pipes"  wrote:

>On 08/18/2013 10:33 PM, Joe Gordon wrote:
>> An alternative I think would be better would be to scrap the
>>use of
>> the SQLAlchemy ORM; keep using the DB engine abstraction
>>support.
>>
>> +1, I am hoping this will provide noticeable performance benefits while
>> being agnostic of what DB back-end is being used.  With the way we use
>>   SQLALchemy being 25x slower then MySQL we have lots of room for
>> improvement (see http://paste.openstack.org/show/44143/ from
>> https://bugs.launchpad.net/nova/+bug/1212418).
>
>@require_admin_context
>def compute_node_get_all(context):
> return model_query(context, models.ComputeNode).\
> options(joinedload('service')).\
> options(joinedload('stats')).\
> all()
>
>Well, yeah... I suppose if you are attempting to create 115K objects in
>memory in Python (Need to collate each ComputeNode model object and each
>of its relation objects for Service and Stats) you are going to run into
>some performance problems. :)
>
>Would be interesting to see what the performance difference would be if
>you instead had dicts instead of model objects and did something like
>this instead (code not tested, just off top of head...):
>
># Assume a method to_dict() that takes a Model
># and returns a dict with appropriate empty dicts for
># relationship fields.
>
>qr = session.query(ComputeNode).join(Service).join(Stats)
>
>results = {}
>
>for record in qr:
>   node_id = record.ComputeNode.id
>   service_id = record.Service.id
>   stat_id = record.ComputeNodeStat.id
>   if node_id not in results.keys():
> results[node_id] = to_dict(record.ComputeNode)
>   if service_id not in results[node_id]['services'].keys():
> results[node_id]['services'][service_id] = to_dict(record.Service)
>   if stat_id not in results[node_id]['stats'].keys():
> results[node_id]['stats'][stat_id] = to_dict(record.ComputeNodeStat)
>
>return results
>
>Whether it would be any faster than SQLAlchemy's joinedload...
>
>Besides that, though, probably is a good idea to look at even the
>existence of DB calls that potentially do that kind of massive query
>returning as A Bad Thing...
>
>Best,
>-jay
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Heat] How the autoscale API should control scaling in Heat

[Christopher Armstrong]

On Fri, Aug 16, 2013 at 1:35 PM, Clint Byrum  wrote:

> Excerpts from Zane Bitter's message of 2013-08-16 09:36:23 -0700:
> > On 16/08/13 00:50, Christopher Armstrong wrote:
> > > *Introduction and Requirements*
> > >
> > > So there's kind of a perfect storm happening around autoscaling in Heat
> > > right now. It's making it really hard to figure out how I should
> compose
> > > this email. There are a lot of different requirements, a lot of
> > > different cool ideas, and a lot of projects that want to take advantage
> > > of autoscaling in one way or another: Trove, OpenShift, TripleO, just
> to
> > > name a few...
> > >
> > > I'll try to list the requirements from various people/projects that may
> > > be relevant to autoscaling or scaling in general.
> > >
> > > 1. Some users want a service like Amazon's Auto Scaling or Rackspace's
> > > Otter -- a simple API that doesn't really involve orchestration.
> > > 2. If such a API exists, it makes sense for Heat to take advantage of
> > > its functionality instead of reimplementing it.
> >
> > +1, obviously. But the other half of the story is that the API is likely
> > be implemented using Heat on the back end, amongst other reasons because
> > that implementation already exists. (As you know, since you wrote it ;)
> >
> > So, just as we will have an RDS resource in Heat that calls Trove, and
> > Trove will use Heat for orchestration:
> >
> >user => [Heat =>] Trove => Heat => Nova
> >
> > there will be a similar workflow for Autoscaling:
> >
> >user => [Heat =>] Autoscaling -> Heat => Nova
> >
>
> After a lot of consideration and an interesting IRC discussion, I think
> the point above makes it clear for me. Autoscaling will have a simpler
> implementation by making use of Heat's orchestration capabilities,
> but the fact that Heat will also use autoscaling is orthogonal to that.
>
> That does beg the question of why this belongs in Heat. Originally
> we had taken the stance that there must be only one control system,
> lest they have a policy-based battle royale. If we only ever let
> autoscaled resources be controlled via Heat (via nested stack produced
> by autoscaling), then there can be only one.. control service (Heat).
>
> By enforcing that autoscaling always talks to "the world" via Heat though,
> I think that reaffirms for me that autoscaling, while not really the same
> project (seems like it could happily live in its own code tree), will
> be best served by staying inside the "OpenStack Orchestration" program.
>
> The question of private RPC or driving it via the API is not all that
> interesting to me. I do prefer the SOA method and having things talk via
> their respective public APIs as it keeps things loosely coupled and thus
> easier to fit into one's brain and debug/change.
>
>
I agree with using only public APIs. I have managed to fit this model of
autoscaling managing a completely independent Heat stack into my brain, and
I am willing to take it and run with it.

Thanks to Zane and Clint for hashing this out with me in a 2-hour IRC
design discussion, it was incredibly helpful :-)

-- 
IRC: radix
Christopher Armstrong
Rackspace
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Blueprint for Amazon VPC API support

[Rudra Rugge]

Hi Salvatore,

Thanks for reviewing. You are right - there are changes planned in Neutron as 
well for the Amazon VPC API support. Here is the link referring to the Neutron 
blueprint.
https://blueprints.launchpad.net/neutron/+spec/juniper-plugin-with-extensions

All calls except related to Instances are implemented as Neutron extensions. In 
addition all the security group APIs and floating IP APIs work in multi mode - 
EC2 mode (like before) and VPC mode (through Neutron).

Regards,
Rudra

On Aug 17, 2013, at 8:06 AM, Salvatore Orlando 
mailto:sorla...@nicira.com>>
 wrote:

Hi Rudra,

In this blueprint what would be the plan for Neutron?
>From the specification it seems the scope is not limited to nova-network, but 
>includes Neutron as well, so you probably have a plan to forward the VPC calls 
>to Neutron.

The question here is whether you've considered implementing the VPC API in 
Neutron directly, with the exception of those calls, like RunInstances which 
pertain to the compute node.

Regards,
Salvatore


On 14 August 2013 20:27, Rudra Rugge 
mailto:rru...@juniper.net>> wrote:
In addition there is one more change:

New euca2ools commands have been added to support all the
VPC related CLI.

Thanks,
Rudra


On Aug 13, 2013, at 10:38 PM, Rudra Rugge 
mailto:rru...@juniper.net>> wrote:

Hi All,

A blueprint has been registered to add Amazon VPC API support.
Currently Amazon EC2 API support already exists in Openstack.
Please review the blueprint and the attached specification. The
specification covers all the VPC APIs and describes how it maps
to the Openstack constructs.

Blueprint
https://blueprints.launchpad.net/nova/+spec/aws-vpc-support

Specification
https://wiki.openstack.org/wiki/Blueprint-aws-vpc-support

The changes are orthogonal to Amazon EC2 APIs but follow the
same model as the EC2 API source code. In addition new unit
tests have been added to cover all the VPC tests.

Please review the blueprint - all comments are welcome.

Thank you,
Rudra


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] General Question about CentOS

[Miller, Mark M (EB SW Cloud - R&D - Corvallis)]

Thank you for the answer and link.

Mark

From: Shake Chen [mailto:shake.c...@gmail.com]
Sent: Friday, August 16, 2013 6:01 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] General Question about CentOS

Now in Centos 6.x ,the Python is 2.6.6, the Openstack can run it. you can check 
the RDO

http://openstack.redhat.com/Quickstart

On Sat, Aug 17, 2013 at 8:05 AM, Yufang Zhang 
mailto:yufang521...@gmail.com>> wrote:
My team has deployed hundreds of compute nodes on CentOS-5.4(with python26 
installed and Xen as hypervisor ) based on Folsom. It does work on our 
production system :)

2013/8/17 Miller, Mark M (EB SW Cloud - R&D - Corvallis) 
mailto:mark.m.mil...@hp.com>>
Is OpenStack supported on CentOS running Python 2.6?

Thanks,

Mark

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



--
Shake Chen
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Murano] Meeting minutes 2013-08-19

[Denis Koryavov]

Hello,

Below, you can see the meeting minutes from today's Murano meeting.

Minutes:
http://eavesdrop.openstack.org/meetings/murano/2013/murano.2013-08-19-15.04.html
Minutes (text):
http://eavesdrop.openstack.org/meetings/murano/2013/murano.2013-08-19-15.04.txt
Log:
http://eavesdrop.openstack.org/meetings/murano/2013/murano.2013-08-19-15.04.log.html


--
Denis
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Stats on blueprint design info / creation times

[Daniel P. Berrange]

In this thread about code review:

  http://lists.openstack.org/pipermail/openstack-dev/2013-August/013701.html

I mentioned that I thought there were too many blueprints created without
sufficient supporting design information and were being used for "tickbox"
process compliance only. I based this assertion on a gut feeling I have
from experiance in reviewing.

To try and get a handle on whether there is truely a problem, I used the
launchpadlib API to extract some data on blueprints [1].

In particular I was interested in seeing:

  - What portion of blueprints have an URL containing an associated
design doc,

  - How long the descriptive text was in typical blueprints

  - Whether a blueprint was created before or after the dev period
started for that major release.


The first two items are easy to get data on. On the second point, I redid
line wrapping on description text to normalize the line count across all
blueprints. This is because many blueprints had all their text on one
giant long line, which would skew results. I thus wrapped all blueprints
at 70 characters.

The blueprint creation date vs release cycle dev start date is a little
harder. I inferred the start date of each release, by using the end date
of the previous release. This is probably a little out but hopefully not
by enough to totally invalidate the usefulness of the stats below. Below,
"Early" means created before start of devel, "Late" means created after
the start of devel period.

The data for the last 3 releases is:

  Series: folsom
Specs: 178
Specs (no URL): 144
Specs (w/ URL): 34
Specs (Early): 38
Specs (Late): 140
Average lines: 5
Average words: 55


  Series: grizzly
Specs: 227
Specs (no URL): 175
Specs (w/ URL): 52
Specs (Early): 42
Specs (Late): 185
Average lines: 5
Average words: 56


  Series: havana
Specs: 415
Specs (no URL): 336
Specs (w/ URL): 79
Specs (Early): 117
Specs (Late): 298
Average lines: 6
Average words: 68


Looking at this data there are 4 key take away points

  - We're creating more blueprints in every release.

  - Less than 1 in 4 blueprints has a link to a design document. 

  - The description text for blueprints is consistently short
(6 lines) across releases.

  - Less than 1 in 4 blueprints is created before the devel
period starts for a release.


You can view the full data set + the script to generate the
data which you can look at to see if I made any logic mistakes:

  http://berrange.fedorapeople.org/openstack-blueprints/


There's only so much you can infer from stats like this, but IMHO think the
stats show that we ought to think about how well we are using blueprints as
design / feature approval / planning tools.


That 3 in 4 blueprint lack any link to a design doc and have only 6 lines of
text description, is a cause for concern IMHO. The blueprints should be giving
code reviewers useful background on the motivation of the dev work & any
design planning that took place. While there are no doubt some simple features
where 6 lines of text is sufficient info in the blueprint, I don't think that
holds true for the majority.

In addition to helping code reviewers, the blueprints are also arguably a
source of info for QA people testing OpenStack and for the docs teams
documenting new features in each release. I'm not convinced that there is
enough info in many of the blueprints to be of use to QA / docs people.


The creation dates of the blueprints are also an interesting data point.
If the design summit is our place for reviewing blueprints and 3 in 4
blueprints in a release are created after the summit, that's alot of
blueprints potentially missing summit discussions. On the other hand many
blueprints will have corresponding discussions on mailing lists too,
which is arguably just as good, or even better than, summit discussions.

Based on the creation dates though & terseness of design info, I think
there is a valid concern here that blueprints are being created just for
reason of "tickbox" process compliance. 

In theory we have an approval process for blueprints, but are we ever
rejecting code submissions for blueprints which are not yet approved ?
I've only noticed that happen a couple of times in Nova for things that
were pretty clearly controversial.

I don't intend to suggest that we have strict rules that all blueprints
must be min X lines of text, or be created by date Y. It is important
to keep the flexibility there to avoid development being drowned in
process without benefits.

I do think we have scope for being more rigourous in our review of
blueprints, asking people to expand on the design info associated with
a blueprint. Perhaps also require that a blueprint is actually approved
by the core team before we go to the trouble of reviewing & approving
the code implementing a blueprint in Gerrit.

Regards,
Daniel

[1] http://berrange.fedorapeople.org/openstack-blueprints/bluepri

Re: [openstack-dev] General Question about CentOS

[Jay Buffington]

On Fri, Aug 16, 2013 at 2:51 PM, Miller, Mark M (EB SW Cloud - R&D -
Corvallis)  wrote:

>   Is OpenStack supported on CentOS running Python 2.6?
>

Oh, I forgot to mention, keystone's py2.6 support seems to currently be
broken because of this bug:
https://bugs.launchpad.net/keystone/+bug/1213284/

Jay
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] General Question about CentOS

[Jay Buffington]

We're doing continuous deployment on Oracle Linux 6.3, using a RHEL kernel
making it almost identical to CentOS 6.3.  We're using keystone, glance,
neutron and nova.  The neutron metadata service doesn't work, nor do
overlapping ips: both need network namespace support in the kernel which
wasn't introduced until 3.0.0.  We use the nova metadata service with an
iptables hack.   We disabled overlapping ips.

There are also some libvirt issues.  We're running the stock libvirt
0.9.10, which, for reasons I can't recall, required running the nova
compute process as root.  Also, there were some issues relating to
libvirt's host-passthrough, which were actually back ported in to the
libvirt 0.9.10 by redhat, but the way nova does the check it doesn't see
this backported capability (see nova's commit 2a236d24a).

Jay


On Fri, Aug 16, 2013 at 2:51 PM, Miller, Mark M (EB SW Cloud - R&D -
Corvallis)  wrote:

>   Is OpenStack supported on CentOS running Python 2.6?
>
> ** **
>
> Thanks,
>
> ** **
>
> Mark
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Help consuming trusts

[Dolph Mathews]

On Mon, Aug 19, 2013 at 6:06 AM, Steven Hardy  wrote:

> On Sun, Aug 18, 2013 at 07:02:04PM +0200, Matthieu Huin wrote:
> > Hi Steve,
> >
> > It might be a bit late for this, but here's a script I wrote when
> experimenting with trusts:
> https://github.com/mhuin/keystone_trust/blob/master/tests/swift_example.sh
> >
> > I hope it'll help you.
>
> Thanks for this!!
>
> Exactly what I was looking for and has enabled me to solve my problem (my
> test code was broken).
>
> I've marked this bug invalid:
>
> https://bugs.launchpad.net/keystone/+bug/1213340
>
> Interestingly, my debugging has highlighted a slightly non-obvious issue
> with
> the creation and consumption of a trust which is probably worth mentioning
> here:
>
> The docs state ""A project_id may not be specified without at least one
> role,
> and vice versa.", however /OS-TRUST/trusts *does* allow you to create a
> trust
> with an empty roles list.
>
> This results in 401 responses whenever you try to consume the trust, which
> is
> not exactly obvious until you realize what's happening..
>
> Can I ask if this is deliberate, or is it a bug in the trusts create code?
>

That certainly sounds like a bug, given that it directly conflicts with the
documented behavior.


>
> It seems odd to allow creation of a trust which is seemingly useless and
> can
> never be consumed?
>

++


>
> Thanks all for your help working through this!
>
> Steve
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Continuous deployment - significant process change

[John Griffith]

On Sun, Aug 18, 2013 at 11:17 PM, Robert Collins
wrote:

> On 19 August 2013 16:15, John Griffith 
> wrote:
>
> > This was pretty well discussed back in April and May IMO.
>
> It petered out with no firm consensus AFAICT. Those of us with CD
> experience are trying to debug the concerns those of us here without
> CD experience have, so that we can bring the benefits in.
>
> > Suffice it to say, I'm very much against the idea of 'disabled features'
> > landing in trunk,
>
> Ok. So you don't like how the Nova V3 API was done then? Or the new
> keystone API? Or is your objection to code
>

Not at all, perhaps I was missing something in your proposal.  If you're
proposing we "do something the way we're already doing it" I've obvioulsy
missed something, or read into things somewhere that perhaps I shouldn't
have.


> > and I'm also not a fan of the idea of an arbitrary max
> > lines of code per patch set.
>
> Once I would have said this, but there is now research into the size
> of change <-> defect rate. The limits proposed are not arbitrary: they
> are evidence based.
>

Just to be clear, I'm not arguing against smaller multiple change-sets, I
believe as do most that these are better for quite a few reasons.  I'm just
saying that I'm not sure about the idea of trying to assign a number right
now.  To be fair it's pretty easy to burn through say 500 lines of code
when implementing a feature in OpenStack, between all the wrappers and
adding unit-tests and hopefully some comments, that diff grows pretty
quickly.


>
> >  A number of folks have pointed out that we're
> > getting better at things like "feature-rush" at the end of a cycle and
> our
> > own community "best practices" enforcement on patch size.
>
> That claim has been made. I think we should do a retrospective after H
> releases to see if it is fact, or wish ;). A growing contributor base
> and long review lead times may counter the policy based approach taken
> in this cycle. Either way we'll still have a freeze period to deal
> with, with it's downsides.
>

Sure, I don't think it's perfect by any means but I do think attempts are
being made.

>
> >  I think that
> > model works well in an Open Source environment, particularly one the
> size of
> > OpenStack with the varied interest and participation.
>
> I think that the model we're converging on for CD in OpenStack will
> work well too; but we need to experiment a bit once the constraints
> are known to really know.
>


>
> > IMO intentionally placing non-working (and thereby useless code as far as
> > I'm concerned) in the project with no testing, no documentation and
> worst of
> > all no guarantee that anybody is ever going to work on said code again
> is a
> > bad idea.
>
> When you say it like that I don't want this either!. Fortunately that
> isn't *at all* what is being proposed.
>

Pheww... ok, perhaps I misunderstood things here, in which case I'll go
back about my business.

>
> - non-working: *none* of the proposals have proposed non-working code.
> - testing: *none* of the proposals have proposed any exception to
> testing policies - all the code should be tested.
> - documentation: Likewise, all the code should be documented, and as
> the feature becomes accessible to early adopters, *obviously* user
> documentation should be present.
> - guarantees that someone will work on a feature: We have no guarantee
> that we'll ever receive another hyper-V patch. In fact, we have no
> guarantees for any feature that anyone will work on it again, whether
> or not the feature is 'complete'. A better question might be: does
> someone with an incomplete feature, or a complete feature, have more
> motivation to keep working on it?
>
> > The explosive growth of what OpenStack is and all of the projects
> > is pretty difficult for folks to get wrapped around already, let alone
> if we
> > start having this unbelievable matrix of flags, paralell features etc.
>
> What new unbelievable matrix are you referring to? We already have a
> huge matrix of features, with no structure or introspection visible,
> and a solid system for managing new ones coming would make landing
> large things like cells much less disruptive.
>
> > Anyway, a number of postings are no longer tracked in this thread it
> seems,
> > but there have been statements from Russell B, Thierry and Michael Still
> > that I strongly agree with here.
>
> What I'm trying to do at this point is capture those concerns
> accurately so they can be addressed. If we're going to have a
> productive step forward on this - e.g. another session at the summit,
> we need to go in with as many concerns captured as possible. Last time
> we had a great session, and then an 'omg what' reaction here, in this
> list.
>

Thanks for the clarification here, I wasn't really sure what your intent
was so I'm glad that you pointed it out here.  Looking forward to the
session on this in 12 weeks and how the conversation proceeds between now
and then.

>
> -Rob
>
>
>
> 

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

[Sandy Walsh]

On 08/18/2013 04:04 PM, Jay Pipes wrote:
> On 08/17/2013 03:10 AM, Julien Danjou wrote:
>> On Fri, Aug 16 2013, Jay Pipes wrote:
>>
>>> Actually, that's the opposite of what I'm suggesting :) I'm suggesting
>>> getting rid of the resource_metadata column in the meter table and
>>> using the
>>> resource table in joins...
>>
>> I think there's a lot of scenario where this would fail, like for
>> example instances being resized; the flavor is a metadata.
> 
> I'm proposing that in these cases, a *new* resource would be added to
> the resource table (and its ID inserted in meter) table with the new
> flavor/instance's metadata.
> 
>> Though, changing the schema to improve performance is a good one, this
>> needs to be thought from the sample sending to the storage, through the
>> whole chain. This is something that will break a lot of current
>> assumption; that doesn't mean it's bad or we can't do it, just that we
>> need to think it through. :)
> 
> Yup, understood completely. The change I am proposing would not affect
> any assumptions made from the point of view of a sample sent to storage.
> The current assumption is that a sample's *exact* state at time of
> sampling would be stored so that the exact sample state could be
> reflected even if the underlying resource that triggered the sample
> changed over time.
> 
> All I am proposing is a change to the existing implementation of that
> assumption: instead of storing the original resource metadata in the
> meter table, we instead ensure that we store the resource in the
> resource table, and upon new sample records being inserted into the
> meter table, we check to see if the resource for the sample is the same
> as it was last time. If it is, we simply insert the resource ID from
> last time. If it isn't, we add a new record to the resource table that
> describes the new resource attributes, and we insert that new resource
> ID into the meter table for that sample...

I'm assuming we wouldn't need a backlink to the older resource?

I'm thinking about how this would work work Events and Request ID's. The
two most common reports we run from StackTach are based on Request ID
and some resource ID.

"Show me all the events related to this Request UUID"
"Show me all the events related to this  UUID"

A new Resource entry would be fine so long as it was still associated
with the underlying Resource UUID (instance, image, etc). We could get
back a list of all the Resources with the same UUID and, if needed,
lookup the metadata for it. This would allow us to see how to the
resource changed over time.

I think that's what you're suggesting ... if so, yep.

As for the first query "... for this Request ID", we'd have to map Event
many related Resources since one event could have a related
instance/image/network/volume/host/scheduler, etc.

These relationships would have to get mapped when the Event is turned
into Meters. Changing the Resource ID might not be a problem if we keep
a common Resource UUID. I have to think about that some more.

Would we use timestamps to determine which Resource is the most recent?


-S




> Best,
> -jay
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

[Sandy Walsh]

On 08/19/2013 09:40 AM, Julien Danjou wrote:
> On Mon, Aug 19 2013, Sandy Walsh wrote:
> 
>> On 08/19/2013 05:08 AM, Julien Danjou wrote:
>>> On Sun, Aug 18 2013, Jay Pipes wrote:
>>>
 I'm proposing that in these cases, a *new* resource would be added to the
 resource table (and its ID inserted in meter) table with the new
 flavor/instance's metadata.
>>>
>>> Ah I see. Considering we're storing metadata as a serialized string
>>> (whereas it's a dict), isn't there a chance we fail?
>>> I'm not sure about the idempotence of the JSON serialization on dicts.
>>
>> Yeah, using a json blob should only be for immutable data. I'm assuming
>> metadata can change so we'd need idempotence. I could easily see two
>> pipelines altering metadata fields. Last write wins. :(
> 
> No, actually I'm not worried about that, it would work as described by
> Jay. It's just that I'm not sure that we can assert json.dumps(somedict)
> returns always the same string.
> 


Gotcha. I think that's fine though. We're going to get that with a
frequently changing resource. So long as we have a common UUID (from the
source system), the database ID can change all it wants.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Flavio Percoco]

On 19/08/13 16:45 +0400, Boris Pavlovic wrote:
Flavio, 


So could you review please patches in Glance? =)



Yes, I'll sync with Mark and other folks to make sure all doubts are
cleared.


--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Boris Pavlovic]

Flavio,

So could you review please patches in Glance? =)


Best regards,
Boris Pavlovic
--
Mirantis Inc.



On Mon, Aug 19, 2013 at 4:33 PM, Flavio Percoco  wrote:

> On 19/08/13 04:33 -0700, Gary Kotton wrote:
>
>> So are you agree with next points?
>> 1) In Havana focus on migrating in all projects to oslo.db code
>>
>>
>> [Gary Kotton] It is worth going for.
>>
>
> +1
>
>
>
>> 2) in IceHouse create and move to oslo.db lib
>>
>> [Gary Kotton] I am in favor of this pending the stability of the oslo db
>> code
>> (which is on the right track)
>>
>>
>>
> I agree with Gary.
>
>
>  And are you agree that we should start working around olso.db lib now.
>>
>> [Gary Kotton] I am not sure what the effort for this is, but if this is
>> just a
>> matter of preparing it all for the start of Icehouse then cool, go for
>> it. I
>> nonetheless suggest speaking with Mark McLoughlinto try and learn lessons
>> from
>> the process with the common config module J
>>
>>
> +1 here as well
>
>
> FF
>
> --
> @flaper87
> Flavio Percoco
>
> __**_
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.**org 
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

[Julien Danjou]

On Mon, Aug 19 2013, Sandy Walsh wrote:

> On 08/19/2013 05:08 AM, Julien Danjou wrote:
>> On Sun, Aug 18 2013, Jay Pipes wrote:
>> 
>>> I'm proposing that in these cases, a *new* resource would be added to the
>>> resource table (and its ID inserted in meter) table with the new
>>> flavor/instance's metadata.
>> 
>> Ah I see. Considering we're storing metadata as a serialized string
>> (whereas it's a dict), isn't there a chance we fail?
>> I'm not sure about the idempotence of the JSON serialization on dicts.
>
> Yeah, using a json blob should only be for immutable data. I'm assuming
> metadata can change so we'd need idempotence. I could easily see two
> pipelines altering metadata fields. Last write wins. :(

No, actually I'm not worried about that, it would work as described by
Jay. It's just that I'm not sure that we can assert json.dumps(somedict)
returns always the same string.

-- 
Julien Danjou
;; Free Software hacker ; freelance consultant
;; http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] VPNaaS API documentation… please review...

[Paul Michali]

https://review.openstack.org/#/c/41702/

Thanks!

PCM (Paul Michali)

MAIL p...@cisco.com
IRC   pcm_  (irc.freenode.net)
TW   @pmichali



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Flavio Percoco]

On 19/08/13 04:33 -0700, Gary Kotton wrote:
So are you agree with next points? 


1) In Havana focus on migrating in all projects to oslo.db code


[Gary Kotton] It is worth going for.


+1



2) in IceHouse create and move to oslo.db lib

[Gary Kotton] I am in favor of this pending the stability of the oslo db code
(which is on the right track)




I agree with Gary.

And are you agree that we should start working around olso.db lib now. 



[Gary Kotton] I am not sure what the effort for this is, but if this is just a
matter of preparing it all for the start of Icehouse then cool, go for it. I
nonetheless suggest speaking with Mark McLoughlinto try and learn lessons from
the process with the common config module J



+1 here as well

FF

--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

[Sandy Walsh]

On 08/19/2013 05:08 AM, Julien Danjou wrote:
> On Sun, Aug 18 2013, Jay Pipes wrote:
> 
>> I'm proposing that in these cases, a *new* resource would be added to the
>> resource table (and its ID inserted in meter) table with the new
>> flavor/instance's metadata.
> 
> Ah I see. Considering we're storing metadata as a serialized string
> (whereas it's a dict), isn't there a chance we fail?
> I'm not sure about the idempotence of the JSON serialization on dicts.

Yeah, using a json blob should only be for immutable data. I'm assuming
metadata can change so we'd need idempotence. I could easily see two
pipelines altering metadata fields. Last write wins. :(

> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Tempest][Neutron] Improving neutron testing

[Salvatore Orlando]

Hi,

I've been trying to spend some time adding a few (hopefully) useful test
scenarios for neutron.
I have never commited anything into tempest, so I would appreciate any
feedback on the approach I am taking.

I have the following goals:
1) enable test_006_check_tenant_network_connectivity for neutron
2) validate tenant network connectivity across a logical router
3) verify default source NAT for outbound traffic
4) verify the extensions for disabling the default source NAT rules

Goals #1 and #2 could have been achieved adding a direct route to the
tenant network from the host running tempest; devstack does something
similar. However this does not really seem the right approach, as in theory
the host running tempest only need to have access to the API endpoints.

So far I have solved this by booting a second server, either in the same
network or in another network reachable via the router, and then pinging
the second server from the first via ssh.

For verifying source NAT behaviour (goals #3 and #4), I am currently using
the first instance to open a tunnel to the second instance, and then
pinging the default gateway from the public network from there. It won't be
correct to perform the ping from the first instance, as that instance is
associated with a floating IP, and therefore the floating IP SNAT rule
would be hit instead of the default one.

Thanks in advance for your advice and feedback.
I shall made WIP patches available soon.

Regards,
Salvatore
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Tempest test failing for neutron

[Rajdeep Dua]

On submitted a Patch for neutron test case
https://review.openstack.org/#/c/42598/


 unrelated tests from tempest seem to be failing.
http://logs.openstack.org/98/42598/2/check/gate-tempest-devstack-vm-neutron-full/4388bc6/console.html

One of the test failure is listed below.

Any idea why this is happening and how to fix this?

==
2013-08-19 11:11:47.424 | FAIL: 
tempest.api.compute.admin.test_fixed_ips.FixedIPsTestJson.test_set_reserve[gate]
2013-08-19 11:11:47.425 | 
tempest.api.compute.admin.test_fixed_ips.FixedIPsTestJson.test_set_reserve[gate]
2013-08-19 11:11:47.425 | 
--
2013-08-19 11:11:47.425 | _StringException: Empty attachments:
2013-08-19 11:11:47.425 |   stderr
2013-08-19 11:11:47.426 |   stdout
2013-08-19 11:11:47.426 | 
2013-08-19 11:11:47.426 | Traceback (most recent call last):
2013-08-19 11:11:47.427 |   File "tempest/api/compute/admin/test_fixed_ips.py", 
line 74, in test_set_reserve
2013-08-19 11:11:47.427 | resp, body = 
self.client.reserve_fixed_ip(self.ip, body)
2013-08-19 11:11:47.427 |   File 
"tempest/services/compute/json/fixed_ips_client.py", line 39, in 
reserve_fixed_ip
2013-08-19 11:11:47.428 | resp, body = self.post(url, json.dumps(body), 
self.headers)
2013-08-19 11:11:47.428 |   File "tempest/common/rest_client.py", line 259, in 
post
2013-08-19 11:11:47.428 | return self.request('POST', url, headers, body)
2013-08-19 11:11:47.428 |   File "tempest/common/rest_client.py", line 387, in 
request
2013-08-19 11:11:47.429 | resp, resp_body)
2013-08-19 11:11:47.429 |   File "tempest/common/rest_client.py", line 432, in 
_error_checker
2013-08-19 11:11:47.429 | raise exceptions.NotFound(resp_body)
2013-08-19 11:11:47.430 | NotFound: Object not found
2013-08-19 11:11:47.430 | Details: {"itemNotFound": {"message": "Fixed IP 
10.1.0.4 not found", "code": 404}}
2013-08-19 11:11:47.430 | 
2013-08-19 11:11:47.431 | ___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Gary Kotton]

From: Boris Pavlovic [mailto:bo...@pavlovic.me]
Sent: Monday, August 19, 2013 2:20 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Proposal oslo.db lib

Flavio,

I'm sorry if I'm being paranoid, I just think we should first focus on
migrating all projects - those that will / should migrate -
successfully and then pulling oslo.db out.

Absolutely agree


Pulling it out during Icehouse won't slow Oslo's db development down.

So the only problem is that is too late for such big changes. Here I am 
absolutely agree. end of H-3 is not the best time =)


So are you agree with next points?
1) In Havana focus on migrating in all projects to oslo.db code

[Gary Kotton] It is worth going for.

2) in IceHouse create and move to oslo.db lib

[Gary Kotton] I am in favor of this pending the stability of the oslo db code 
(which is on the right track)

And are you agree that we should start working around olso.db lib now.

[Gary Kotton] I am not sure what the effort for this is, but if this is just a 
matter of preparing it all for the start of Icehouse then cool, go for it. I 
nonetheless suggest speaking with Mark McLoughlinto try and learn lessons from 
the process with the common config module :)

To be able to start switching to it at begin of IceHouse?

And what about Glance patches ?;)
Еheir are ready since Aug 6, 2013 2:41 PM and there is no reviews at all.


Best regards,
Boris Pavlovic
---
Mirantis Inc.



On Mon, Aug 19, 2013 at 2:52 PM, Flavio Percoco 
mailto:fla...@redhat.com>> wrote:
On 19/08/13 14:17 +0400, Boris Pavlovic wrote:
Flavio,

Agreed. I'd also like to see other project migrated before pulling
oslo.db out from oslo-incubator


as I wrote before oslo.db code is used by:  Nova, Neutron, Cinder, Ironic,
Ceilometer use oslo.db. And we have already patches to switch in Glance to id.
And we are woking in Keystone and Heat.
Why (nova, neutron, cinder, ironic and ceilometer) is not enough to say that
the code is OK?

Because we're talking about the database code here. All projects
depend / will depend on it. I wouldn't consider the whole API stable
until all projects have successfully migrated to it. There are corner
cases that could have been missed and that would require a new oslo.db
release - assuming it has been pulled out - for them to be applied to
projects using it.

I'm sorry if I'm being paranoid, I just think we should first focus on
migrating all projects - those that will / should migrate -
successfully and then pulling oslo.db out.

Hm, I really don't see big problems with migrating to oslo.db lib even in this
moment, because in oslo-incubator and in oslo.db is the same code.. Could you
explain what problem you see in process of migartion? (For me it is add one
more requirments, remove openstack/db/.. folder, switch imports)..

AFAIK, doing this will require to:

1) Set up a new repository
2) Make a release / tarball that you could use in the requirements
file.
3) Change imports on all projects using oslo-incubator db code.

All the steps above require people to review patches and step 3 needs
to be done on 5 projects. In the meantime, you're also submitting
patches to the not-yet migrated projects that are based on Oslo's code
and those patches, if not yet merged - will also have to be updated.

My concern is that there's a lot of work yet to be done on this area
and I don't think it is ready to be pulled out yet. I'd prefer a
migration process similar to oslo.config. All project were using it
before it was pulled out and that was done when the API was considered
stable.

Pulling it out during Icehouse won't slow Oslo's db development down.
Remaining projects can still be migrated and released during Havanna.

Cheers,

FF

--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Boris Pavlovic]

Flavio,

I'm sorry if I'm being paranoid, I just think we should first focus on
migrating all projects - those that will / should migrate -
successfully and then pulling oslo.db out.

Absolutely agree


Pulling it out during Icehouse won't slow Oslo's db development down.

So the only problem is that is too late for such big changes. Here I am
absolutely agree. end of H-3 is not the best time =)


So are you agree with next points?
1) In Havana focus on migrating in all projects to oslo.db code
2) in IceHouse create and move to oslo.db lib

And are you agree that we should start working around olso.db lib now.
To be able to start switching to it at begin of IceHouse?

And what about Glance patches ?;)
Еheir are ready since Aug 6, 2013 2:41 PM and there is no reviews at all.


Best regards,
Boris Pavlovic
---
Mirantis Inc.




On Mon, Aug 19, 2013 at 2:52 PM, Flavio Percoco  wrote:

> On 19/08/13 14:17 +0400, Boris Pavlovic wrote:
>
>> Flavio,
>>
>> Agreed. I'd also like to see other project migrated before pulling
>> oslo.db out from oslo-incubator
>>
>>
>> as I wrote before oslo.db code is used by:  Nova, Neutron, Cinder, Ironic,
>> Ceilometer use oslo.db. And we have already patches to switch in Glance
>> to id.
>> And we are woking in Keystone and Heat.
>> Why (nova, neutron, cinder, ironic and ceilometer) is not enough to say
>> that
>> the code is OK?
>>
>
> Because we're talking about the database code here. All projects
> depend / will depend on it. I wouldn't consider the whole API stable
> until all projects have successfully migrated to it. There are corner
> cases that could have been missed and that would require a new oslo.db
> release - assuming it has been pulled out - for them to be applied to
> projects using it.
>
> I'm sorry if I'm being paranoid, I just think we should first focus on
> migrating all projects - those that will / should migrate -
> successfully and then pulling oslo.db out.
>
>
>  Hm, I really don't see big problems with migrating to oslo.db lib even in
>> this
>> moment, because in oslo-incubator and in oslo.db is the same code.. Could
>> you
>> explain what problem you see in process of migartion? (For me it is add
>> one
>> more requirments, remove openstack/db/.. folder, switch imports)..
>>
>
> AFAIK, doing this will require to:
>
> 1) Set up a new repository
> 2) Make a release / tarball that you could use in the requirements
> file.
> 3) Change imports on all projects using oslo-incubator db code.
>
> All the steps above require people to review patches and step 3 needs
> to be done on 5 projects. In the meantime, you're also submitting
> patches to the not-yet migrated projects that are based on Oslo's code
> and those patches, if not yet merged - will also have to be updated.
>
> My concern is that there's a lot of work yet to be done on this area
> and I don't think it is ready to be pulled out yet. I'd prefer a
> migration process similar to oslo.config. All project were using it
> before it was pulled out and that was done when the API was considered
> stable.
>
> Pulling it out during Icehouse won't slow Oslo's db development down.
> Remaining projects can still be migrated and released during Havanna.
>
> Cheers,
>
> FF
>
> --
> @flaper87
> Flavio Percoco
>
> __**_
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.**org 
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Help consuming trusts

[Steven Hardy]

On Sun, Aug 18, 2013 at 07:02:04PM +0200, Matthieu Huin wrote:
> Hi Steve,
> 
> It might be a bit late for this, but here's a script I wrote when 
> experimenting with trusts: 
> https://github.com/mhuin/keystone_trust/blob/master/tests/swift_example.sh
> 
> I hope it'll help you.

Thanks for this!!

Exactly what I was looking for and has enabled me to solve my problem (my test 
code was broken).

I've marked this bug invalid:

https://bugs.launchpad.net/keystone/+bug/1213340

Interestingly, my debugging has highlighted a slightly non-obvious issue with
the creation and consumption of a trust which is probably worth mentioning here:

The docs state ""A project_id may not be specified without at least one role,
and vice versa.", however /OS-TRUST/trusts *does* allow you to create a trust
with an empty roles list.

This results in 401 responses whenever you try to consume the trust, which is
not exactly obvious until you realize what's happening..

Can I ask if this is deliberate, or is it a bug in the trusts create code?

It seems odd to allow creation of a trust which is seemingly useless and can
never be consumed?

Thanks all for your help working through this!

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Boris Pavlovic]

Mark,

Main part of oslo is:
1) common migration testing
2) common sqla.models
3) common hacks around sqla and sqla-migrate
4) common work around engines and sessions


All these points are implemented in Glance almost in the same way as in
Oslo.
Also we are able to use only part of this code in Glance, and add some
other things that are glance related over this code.

Our current 2 patches on review do next things:
1) Copy paste oslo.db code into glance
2) Use sqla session/engine/exception wrappers
3) Remove Glance code that covers session/engine/exception

So I really don't see any bad thing in this code:
1) If you would like to implement other backends => this change won't block
it
2) If you would like to make some other sqla utitlites or glance related
things => this change won't block it
3) If there are bugs => fix it in oslo and sync => this change won't block
it

 So I really don't see any reason to block work around migration to oslo.db
code in Glance.


Best regards,
Boris Pavlovic
---
Mirantis Inc.




On Fri, Aug 16, 2013 at 10:41 PM, Mark Washenberger <
mark.washenber...@markwash.net> wrote:

> I would prefer to pick and choose which parts of oslo common db code to
> reuse in glance. Most parts there look great and very useful. However, some
> parts seem like they would conflict with several goals we have.
>
> 1) To improve code sanity, we need to break away from the idea of having
> one giant db api interface
> 2) We need to improve our position with respect to new, non SQL drivers
> - mostly, we need to focus first on removing business logic
> (especially authz) from database driver code
> - we also need to break away from the strict functional interface,
> because it limits our ability to express query filters and tends to lump
> all filter handling for a given function into a single code block (which
> ends up being defect-rich and confusing as hell to reimplement)
> 3) It is unfortunate, but I must admit that Glance's code in general is
> pretty heavily coupled to the database code and in particular the schema.
> Basically the only tool we have to manage that problem until we can fix it
> is to try to be as careful as possible about how we change the db code and
> schema. By importing another project, we lose some of that control. Also,
> even with the copy-paste model for oslo incubator, code in oslo does have
> some of its own reasons to change, so we could potentially end up in a
> conflict where glance db migrations (which are operationally costly) have
> to happen for reasons that don't really matter to glance.
>
> So rather than framing this as "glance needs to use oslo common db code",
> I would appreciate framing it as "glance database code should have features
> X, Y, and Z, some of which it can get by using oslo code." Indeed, I
> believe in IRC we discussed the idea of writing up a wiki listing these
> feature improvements, which would allow a finer granularity for evaluation.
> I really prefer that format because it feels more like planning and less
> like debate :-)
>
>  I have a few responses inline below.
>
> On Fri, Aug 16, 2013 at 6:31 AM, Victor Sergeyev 
> wrote:
>
>> Hello All.
>>
>> Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some
>> questions about Oslo DB code, and why is it so important to use it instead
>> of custom implementation and so on. As there were a lot of questions it was
>> really hard to answer on all this questions in IRC. So we decided that
>> mailing list is better place for such things.
>>
>> List of main questions:
>>
>> 1. What includes oslo DB code?
>> 2. Why is it safe to replace custom implementation by Oslo DB code?
>> 3. Why oslo DB code is better than custom implementation?
>> 4. Why oslo DB code won’t slow up project development progress?
>> 5. What we are going actually to do in Glance?
>> 6. What is the current status?
>>
>> Answers:
>>
>> 1. What includes oslo DB code?
>>
>> Currently Oslo code improves different aspects around DB:
>> -- Work with SQLAlchemy models, engine and session
>> -- Lot of tools for work with SQLAlchemy
>>
> -- Work with unique keys
>> -- Base test case for work with database
>> -- Test migrations against different backends
>> -- Sync DB Models with actual schemas in DB (add test that they are
>> equivalent)
>>
>>
>> 2. Why is it safe to replace custom implementation by Oslo DB code?
>>
>> Oslo module, as base openstack module, takes care about code quality.
>> Usually, common code more readable (most of flake8 checks enabled in Oslo)
>> and have better test coverage.  Also it was tested in different use-cases
>> (in production also) in an other projects so bugs in Oslo code were already
>> fixed. So we can be sure, that we use high-quality code.
>>
>
> Alas, while testing and static style analysis are important, they are not
> the only relevant aspects of code quality. Architectural choices are also
> relevant. The best reusable code places few requirements on the code that
> reuses

Re: [openstack-dev] Proposal oslo.db lib

[Flavio Percoco]

On 19/08/13 14:17 +0400, Boris Pavlovic wrote:

Flavio,

Agreed. I'd also like to see other project migrated before pulling
oslo.db out from oslo-incubator


as I wrote before oslo.db code is used by:  Nova, Neutron, Cinder, Ironic,
Ceilometer use oslo.db. And we have already patches to switch in Glance to id.
And we are woking in Keystone and Heat. 


Why (nova, neutron, cinder, ironic and ceilometer) is not enough to say that
the code is OK?


Because we're talking about the database code here. All projects
depend / will depend on it. I wouldn't consider the whole API stable
until all projects have successfully migrated to it. There are corner
cases that could have been missed and that would require a new oslo.db
release - assuming it has been pulled out - for them to be applied to
projects using it.

I'm sorry if I'm being paranoid, I just think we should first focus on
migrating all projects - those that will / should migrate -
successfully and then pulling oslo.db out.


Hm, I really don't see big problems with migrating to oslo.db lib even in this
moment, because in oslo-incubator and in oslo.db is the same code.. Could you
explain what problem you see in process of migartion? (For me it is add one
more requirments, remove openstack/db/.. folder, switch imports)..


AFAIK, doing this will require to:

1) Set up a new repository
2) Make a release / tarball that you could use in the requirements
file.
3) Change imports on all projects using oslo-incubator db code.

All the steps above require people to review patches and step 3 needs
to be done on 5 projects. In the meantime, you're also submitting
patches to the not-yet migrated projects that are based on Oslo's code
and those patches, if not yet merged - will also have to be updated.

My concern is that there's a lot of work yet to be done on this area
and I don't think it is ready to be pulled out yet. I'd prefer a
migration process similar to oslo.config. All project were using it
before it was pulled out and that was done when the API was considered
stable.

Pulling it out during Icehouse won't slow Oslo's db development down.
Remaining projects can still be migrated and released during Havanna.

Cheers,
FF

--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Can I Expose the host-aggregate as availability zone?

[sudheesh sk]

Hi,


1) Can I Expose
the host-aggregate as availability zone?


2) Is there anyway to make the Host Aggregate dynamically growing (with each VM 
creation- add the host to host aggregate if its not already there)?

Thanks,
Sudheesh___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Can I Expose the host-aggregate as availability zone?

[sudheesh sk]

Hi,

1) Can I Expose
the host-aggregate as availability zone?


2) Is there anyway to make the Host Aggregate dynamically growing (with each VM 
creation- add the host to host aggregate if its not already there)?

Thanks,
Sudheesh___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Boris Pavlovic]

Flavio,

Agreed. I'd also like to see other project migrated before pulling
oslo.db out from oslo-incubator


as I wrote before oslo.db code is used by:  Nova, Neutron, Cinder, Ironic,
Ceilometer use oslo.db. And we have already patches to switch in Glance to
id. And we are woking in Keystone and Heat.

Why (nova, neutron, cinder, ironic and ceilometer) is not enough to say
that the code is OK?


I'd also add that pulling oslo db into its own package means that
projects using Oslo's db code have to be migrated as well. I think
it's a bit late for that. The focus should be on making sure current
code is stable enough for the not-so-far release.


Hm, I really don't see big problems with migrating to oslo.db lib even in
this moment, because in oslo-incubator and in oslo.db is the same code..
Could you explain what problem you see in process of migartion? (For me it
is add one more requirments, remove openstack/db/.. folder, switch
imports)..



On Mon, Aug 19, 2013 at 1:29 PM, Flavio Percoco  wrote:

> On 19/08/13 00:34 -0700, Gary Kotton wrote:
>
>> Hi,
>>
>> I have a number of things to say here:
>>
>> 1.   Great work in getting the DB into the common and ironing out the
>> issues
>>
>>
> +1
>
>
>  2.   As far as I know only Neutron and Nova are making use of the
>> common DB
>> code. Neutron has been using this since the beginning of H2 (this did not
>> resolve all of the issues that we had and we had) and Nova has just
>> recently
>> upgraded to the latest DB code (this was a few weeks ago).
>>
>> 3.In general I like the idea of having a separate lib for this
>> but have
>> a number of reservations regarding the timing and stability:
>>
>> a.   I do not think that this has been running long enough in Neutron
>> and
>> Nova for us to give it a stamp of approval (the common CFG code was at
>> least
>> one cycle as common code prior to moving into its own lib). I think that
>> in
>> Neutron we still have a number of issues with load on the DB. I need to
>> double
>> check on this.
>>
>
> Agreed. I'd also like to see other project migrated before pulling
> oslo.db out from oslo-incubator
>
>
>
>> b.  I think that the beginning of Icehouse is a good time. When we
>> moved to
>> the CFG library there were a number of hickups and issues along the way. I
>> think that Mark (oslo PTL) can elaborate a little more on this. Timing is
>> essential.
>>
>
> +1
>
> I'd also add that pulling oslo db into its own package means that
> projects using Oslo's db code have to be migrated as well. I think
> it's a bit late for that. The focus should be on making sure current
> code is stable enough for the not-so-far release.
>
> Thanks for the hard work!
> FF
>
> --
> @flaper87
> Flavio Percoco
>
>
> __**_
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.**org 
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Flavio Percoco]

On 19/08/13 00:34 -0700, Gary Kotton wrote:

Hi,

I have a number of things to say here:

1.   Great work in getting the DB into the common and ironing out the
issues



+1


2.   As far as I know only Neutron and Nova are making use of the common DB
code. Neutron has been using this since the beginning of H2 (this did not
resolve all of the issues that we had and we had) and Nova has just recently
upgraded to the latest DB code (this was a few weeks ago).

3.In general I like the idea of having a separate lib for this but have
a number of reservations regarding the timing and stability:

a.   I do not think that this has been running long enough in Neutron and
Nova for us to give it a stamp of approval (the common CFG code was at least
one cycle as common code prior to moving into its own lib). I think that in
Neutron we still have a number of issues with load on the DB. I need to double
check on this.


Agreed. I'd also like to see other project migrated before pulling
oslo.db out from oslo-incubator



b.  I think that the beginning of Icehouse is a good time. When we moved to
the CFG library there were a number of hickups and issues along the way. I
think that Mark (oslo PTL) can elaborate a little more on this. Timing is
essential.


+1

I'd also add that pulling oslo db into its own package means that
projects using Oslo's db code have to be migrated as well. I think
it's a bit late for that. The focus should be on making sure current
code is stable enough for the not-so-far release.

Thanks for the hard work!
FF

--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

[Daniel P. Berrange]

On Mon, Aug 19, 2013 at 01:28:25AM -0700, Tim Smith wrote:
> On Sun, Aug 18, 2013 at 1:28 PM, Robert Collins
> wrote:
> 
> > On 17 August 2013 07:01, Russell Bryant  wrote:
> >
> > >> Maybe we've grown up to the point where we have to be more careful and
> > >> not introduce
> > >> these kind of features and the maintenance cost of introducing
> > >> experimental features is
> > >> too great. If that is the community consensus, then I'm happy keep the
> > >> live snapshot stuff
> > >> in a branch on github for people to experiment with.
> > >
> > > My feeling after following this discussion is that it's probably best to
> > > keep baking in another branch (github or whatever).  The biggest reason
> > > is because of the last comment quoted from Daniel Berrange above.  I
> > > feel that like that is a pretty big deal.
> >
> > So, reading between the lines here, I guess you're worried that we'd
> > let code paths that violate what upstream will support leak into the
> > main codepaths for libvirt - and thus we'd end up with a situation
> > where we aren't supported by upstream for all regular operations.
> >
> > I agree that *that* is a big deal : is there something we could do to
> > prevent that happening? E.g. annotating this whole thing as
> > experimental/not upstream supported or something?
> >
> 
> My understanding is that the live-snapshot extensions are to be disabled by
> default (either on the driver side or the API side) and must be explicitly
> enabled via a nova configuration change. Thus, it is not interferant with
> mainline codepaths by default, and the user will not place themself into an
> "unmaintainable" position unless they or their OpenStack distro provider
> flips that particular switch.
>
> That would seem to me a sufficiently high bar for preventing the user from
> shooting themselves in the foot. As for any objections to the mere
> _existence_ of the live-snapshot capability, I propose that the market
> answer the question as to whether or not the feature has value.

NB live-snapshots are fine if used for rollback/forward of an individual
VMs. It is only use of live snapshots for the purpose of cloning VMs
that's the issue here. Whether the market wants live cloning or not is
irrelevant until a vendor will actually support it from a technology POV

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

[Daniel P. Berrange]

On Mon, Aug 19, 2013 at 08:28:58AM +1200, Robert Collins wrote:
> On 17 August 2013 07:01, Russell Bryant  wrote:
> 
> >> Maybe we've grown up to the point where we have to be more careful and
> >> not introduce
> >> these kind of features and the maintenance cost of introducing
> >> experimental features is
> >> too great. If that is the community consensus, then I'm happy keep the
> >> live snapshot stuff
> >> in a branch on github for people to experiment with.
> >
> > My feeling after following this discussion is that it's probably best to
> > keep baking in another branch (github or whatever).  The biggest reason
> > is because of the last comment quoted from Daniel Berrange above.  I
> > feel that like that is a pretty big deal.
> 
> So, reading between the lines here, I guess you're worried that we'd
> let code paths that violate what upstream will support leak into the
> main codepaths for libvirt - and thus we'd end up with a situation
> where we aren't supported by upstream for all regular operations.

Yes, if you perform a live clone of a VM, then you have effectively
tainted that VM for the rest of its lifetime. From the virt host
vendors' POV, any unexpected or problematic behaviour you get from
that VM thereafter will be outside scope of support. The onus would
be on the openstack sysadmin to demonstrate that the same problem
occurs without the use of live cloning.

Running a production cloud using a feature that your virt host
vendor considers unsupported would be somewhat reckless IMHO, unless
you think your sysadmins think they have the skills to solve all
possible problems in that area themselves, which is unlikely for most
cloud vendors.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Flavio Percoco]

On 19/08/13 02:51 -0400, Jay Pipes wrote:

On 08/19/2013 12:56 AM, Joshua Harlow wrote:

Another good article from an ex-coworker that keeps on making more and
more sense the more projects I get into...

http://seldo.com/weblog/2011/08/11/orm_is_an_antipattern

Your mileage/opinion though may vary :)


2) I highly caution folks who think a No-SQL store is a good storage 
solution for any of the data currently used by Nova, Glance 
(registry), Cinder (registry), Ceilometer, and Quantum. All of the 
data stored and manipulated in those projects is HIGHLY relational 
data, and not objects/documents. Switching to use a KVS for highly 
relational data is a terrible decision. You will just end up 
implementing joins in your code...




+1

FWIW, I'm a huge fan of NoSQL technologies but I couldn't agree more
here.

FF

--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Flavio Percoco]

On 18/08/13 18:47 -0400, Jay Pipes wrote:

On 08/18/2013 06:28 PM, Joe Gordon wrote:


On Aug 18, 2013 3:58 PM, "Jay Pipes" mailto:jaypi...@gmail.com>> wrote:
>
> On 08/18/2013 03:53 AM, Joshua Harlow wrote:
>>
>> I always just liked SQL as the database abstraction layer ;)
>>
>> On a more serious note I think novas new object model might be a way
to go but in all honesty there won't be a one size fits all solution. I
just don't think sqlalchemy is that solution personally (maybe if we
just use sqlalchemy core it will be better and eject just the orm layer).
>
>
> What is specifically wrong with SQLAlchemy's ORM layer? What would
you replace it with? Why would use SQLAlchemy's "core" be better?
>
> I've seen little evidence that SQLAlchemy's ORM layer is the cause
for database performance problems. Rather, I've found that the database
schemas in use -- and in some cases, the *way* that the SQLAlchemy ORM
is called (for example, doing correlated subqueries instead of straight
joins) -- are primary causes for database performance issues.

From what I have seen the issue is both the queries and the ORM layer.
See https://bugs.launchpad.net/nova/+bug/1212418  for details.


Good point.

For the record, I'm not a fan of lazy/eager loading of relations in 
the models themselves, but instead always being explicit about the 
exact data you wish to query for.


It's similar in nature to the SQL best practice of never doing SELECT 
* FROM  and instead of always being explicity about the columns 
you wish to retrieve...




+1

I've seen a couple of cases where this is not being taken under
consideration. I'd like to see some of the lazy loaded relations being
explicitly loaded. 


I think a good rule for this is:

"If you know you'll need it, then load it. If you don't know it, then
you're *probably* doing something wrong."

Cheers,
FF

--
@flaper87
Flavio Percoco

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] live-snapshot/cloning of virtual machines

[Tim Smith]

On Sun, Aug 18, 2013 at 1:28 PM, Robert Collins
wrote:

> On 17 August 2013 07:01, Russell Bryant  wrote:
>
> >> Maybe we've grown up to the point where we have to be more careful and
> >> not introduce
> >> these kind of features and the maintenance cost of introducing
> >> experimental features is
> >> too great. If that is the community consensus, then I'm happy keep the
> >> live snapshot stuff
> >> in a branch on github for people to experiment with.
> >
> > My feeling after following this discussion is that it's probably best to
> > keep baking in another branch (github or whatever).  The biggest reason
> > is because of the last comment quoted from Daniel Berrange above.  I
> > feel that like that is a pretty big deal.
>
> So, reading between the lines here, I guess you're worried that we'd
> let code paths that violate what upstream will support leak into the
> main codepaths for libvirt - and thus we'd end up with a situation
> where we aren't supported by upstream for all regular operations.
>
> I agree that *that* is a big deal : is there something we could do to
> prevent that happening? E.g. annotating this whole thing as
> experimental/not upstream supported or something?
>

My understanding is that the live-snapshot extensions are to be disabled by
default (either on the driver side or the API side) and must be explicitly
enabled via a nova configuration change. Thus, it is not interferant with
mainline codepaths by default, and the user will not place themself into an
"unmaintainable" position unless they or their OpenStack distro provider
flips that particular switch.

That would seem to me a sufficiently high bar for preventing the user from
shooting themselves in the foot. As for any objections to the mere
_existence_ of the live-snapshot capability, I propose that the market
answer the question as to whether or not the feature has value.

Cheers,
Tim




> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] Proposal: Get rid of deleted column

[Julien Danjou]

On Mon, Aug 19 2013, Jay Pipes wrote:

> Thoughts?

+1

-- 
Julien Danjou
;; Free Software hacker ; freelance consultant
;; http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Julien Danjou]

On Mon, Aug 19 2013, Jay Pipes wrote:

> 2) I highly caution folks who think a No-SQL store is a good storage
> solution for any of the data currently used by Nova, Glance (registry),
> Cinder (registry), Ceilometer, and Quantum. All of the data stored and
> manipulated in those projects is HIGHLY relational data, and not
> objects/documents. Switching to use a KVS for highly relational data is a
> terrible decision. You will just end up implementing joins in your code...

+1000.

-- 
Julien Danjou
# Free Software hacker # freelance consultant
# http://julien.danjou.info


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Ceilometer] Concerning get_resources/get_meters and the Ceilometer API

[Julien Danjou]

On Sun, Aug 18 2013, Jay Pipes wrote:

> I'm proposing that in these cases, a *new* resource would be added to the
> resource table (and its ID inserted in meter) table with the new
> flavor/instance's metadata.

Ah I see. Considering we're storing metadata as a serialized string
(whereas it's a dict), isn't there a chance we fail?
I'm not sure about the idempotence of the JSON serialization on dicts.

-- 
Julien Danjou
/* Free Software hacker * freelance consultant
   http://julien.danjou.info */


signature.asc
Description: PGP signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] Proposal: Get rid of deleted column

[Boris Pavlovic]

Hi Jay,


When I started working around unique keys, I tried to use deleted_at
column. so answer about why we don't use deleted_at column you could read
in Devananda's comment on my patch https://review.openstack.org/#/c/16162/ .

Also I should mention that this is really huge change and it will take a
lot of time to implement it.  E.g. I started working around unique keys at
begin of the Grizzly and we still didn't finish this work in OpenStack
projects (except Nova where we have last the patch on review
https://review.openstack.org/#/c/36880/).


Best regards,
Boris Pavlovic
--
Mirantis Inc.







On Mon, Aug 19, 2013 at 11:39 AM, Jay Pipes  wrote:

> I'm throwing this up here to get some feedback on something that's always
> bugged me about the model base used in many of the projects.
>
> There's a mixin class that looks like so:
>
> class SoftDeleteMixin(object):
> deleted_at = Column(DateTime)
> deleted = Column(Integer, default=0)
>
> def soft_delete(self, session=None):
> """Mark this object as deleted."""
> self.deleted = self.id
> self.deleted_at = timeutils.utcnow()
> self.save(session=session)
>
> Once mixed in to a concrete model class, the primary join is typically
> modified to include the deleted column, like so:
>
> class ComputeNode(BASE, NovaBase):
> ...
> service = relationship(Service,
>backref=backref('compute_node'**),
>foreign_keys=service_id,
>primaryjoin='and_('
> 'ComputeNode.service_id == Service.id,'
> 'ComputeNode.deleted == 0)')
>
> My proposal is to get rid of the deleted column in the SoftDeleteMixin
> class entirely, as it is redundant with the deleted_at column. Instead of
> doing a join condition on deleted == 0, one would instead just do the join
> condition on deleted_at is None, which translates to the SQL: AND
> deleted_at IS NULL.
>
> There isn't much of a performance benefit -- you're only reducing the row
> size by 4 bytes. But, you'd remove the redundant data from all the tables,
> which would make the normal form freaks like myself happy ;)
>
> Thoughts?
>
> -jay
>
> __**_
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.**org 
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db] Proposal: Get rid of deleted column

[Chris Behrens]

'deleted' is used so that we can have proper unique constraints by setting it 
to `id` on deletion.  This was not the case until Grizzly, and before Grizzly I 
would have agreed completely.

- Chris

On Aug 19, 2013, at 12:39 AM, Jay Pipes  wrote:

> I'm throwing this up here to get some feedback on something that's always 
> bugged me about the model base used in many of the projects.
> 
> There's a mixin class that looks like so:
> 
> class SoftDeleteMixin(object):
>deleted_at = Column(DateTime)
>deleted = Column(Integer, default=0)
> 
>def soft_delete(self, session=None):
>"""Mark this object as deleted."""
>self.deleted = self.id
>self.deleted_at = timeutils.utcnow()
>self.save(session=session)
> 
> Once mixed in to a concrete model class, the primary join is typically 
> modified to include the deleted column, like so:
> 
> class ComputeNode(BASE, NovaBase):
>...
>service = relationship(Service,
>   backref=backref('compute_node'),
>   foreign_keys=service_id,
>   primaryjoin='and_('
>'ComputeNode.service_id == Service.id,'
>'ComputeNode.deleted == 0)')
> 
> My proposal is to get rid of the deleted column in the SoftDeleteMixin class 
> entirely, as it is redundant with the deleted_at column. Instead of doing a 
> join condition on deleted == 0, one would instead just do the join condition 
> on deleted_at is None, which translates to the SQL: AND deleted_at IS NULL.
> 
> There isn't much of a performance benefit -- you're only reducing the row 
> size by 4 bytes. But, you'd remove the redundant data from all the tables, 
> which would make the normal form freaks like myself happy ;)
> 
> Thoughts?
> 
> -jay
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Jay Pipes]

OK, cool. I'm in agreement with your explained storage/logic separation 
below.


Cheers,
-jay

On 08/19/2013 03:12 AM, Robert Collins wrote:

On 19 August 2013 18:35, Jay Pipes  wrote:


http://www.codinghorror.com/blog/2006/06/object-relational-mapping-is-the-vietnam-of-computer-science.html

There is no proper use of an ORM.



I'm not a super-fan of ORMs, Robert. I'm not sure why you're insisting on
taking me down this road...


Sorry, not sure how we ended up here ;)


All I'm saying is that we should be careful not to swap one set of problems
for another. I say this because I've seen the Nova data-access code develop
from its very earliest days, up to this point. I've seen the horrors of
trying to mask an object approach on top of a non-relational data store,
witnessed numerous attempts to rewrite the way that connection pooling and
session handling is done, and in general just noticed the tension between
the two engineering factions that want to keep things agnostic towards
backend storage and at the same time make the backend storage perform and
scale adequately.


Ah! Ok, completely agree: playing flip-flop on problem sets would be a
poor outcome.


I'm not sure why you are being so aggressive about this topic. I certainly
am not being aggressive about my responses -- just cautioning that the
existing codebase has seen its fair share of refactoring, some of which has
been a failure and had to be reverted. I would hate to jump into a frenzy to
radically change the way that the data access code works in Nova without a
good discussion.


I didn't intend to be aggressive - sorry - super sorry in fact. I've
been burnt by months of effort turning around problem codebases where
the ORM was a significant cause of the problems.



But then I guarantee somebody is gonna spend a bunch of time writing an
object-oriented API to the model objects because the ORM is very useful
for
the data modification part of the DB interaction.



!cite - seriously...



? I give an example below... a cautionary tale if you will, about one
possible consequence of "getting rid of the ORM".


I think what I really meant here is 'you say months, but if we're
writing an object-orientated API surely we'd just use one of the
mapping techniques available in SQLAlchemy..'


This strawman is one way that it might be written. Given that a
growing set of our projects have non-SQL backends, this doesn't look
like the obvious way to phrase it to me.



I'm using the SQLAlchemy Core API above, with none of the SQLAlchemy ORM
code... which is (I thought), what you were proposing we do? How is that a
strawman argument? :(


So what is in my head is that we have two layers:
business logic
storage logic

And the thing I don't like about the ORM approach is that our business
logic objects are storage logic objects - even though we don't use
http://martinfowler.com/eaaCatalog/domainModel.html we can easily
trigger late evaluation when traversing collections. In particular
because we have large numbers of developers who are likely going to
not be holding the entire problem domain in their head; the churn that
results on code and design tends to throw things out again and again
over time. And we have IMO too much business logic in the
db/sqlalchemy/api.py files scattered around.

So, what I'd like to see is something where the storage layer and
logic layer are more thoroughly decoupled: only return plain ol Python
objects from the DB layer; but within that layer I wouldn't object to
an ORM being used; secondly I'd like to make sure we don't end up
making business decisions in the storage layer, because that makes it
harder when porting to a different storage layer - such as the nova
conductor is.

So the business logic layer for adding a fixed IP would be something like:
i = business.Instance.find(blah=blah)
ip = business.FixedIP(blah=blah)
i.fixed_ips.append(ip)
storage.save(i)

i and ip would be plain ol python objects
storage.save would have the same semantics as an RPC call - it could
do a transaction itself, but there's no holding transactions between
calls to save.

This is very close to:


instead of this:

i = Instance(blah=blah)
ip = FixedIp(blah=blah)
i.fixed_ips.append(ip)
session.add(u)
session.commit()


But there is no ORM exposed to the developers working with the storage
API - it's contained.


And so you've thrown the baby out with the bathwater and made more work
for
everyone.



Perhaps; or perhaps we've avoided a raft of death-by-thousand-cuts
bugs across the project.



Could just as easily introduce the same bugs by radically redesigning the
data access code without first considering all sides of the problem domain.


Totally!

Again, sorry for the tone before, I can only claim a) been burnt in
the past and and b) a week or so of reduced sleep thanks to baby :(.

-Rob




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/list

[openstack-dev] [oslo.db] Proposal: Get rid of deleted column

[Jay Pipes]

I'm throwing this up here to get some feedback on something that's 
always bugged me about the model base used in many of the projects.


There's a mixin class that looks like so:

class SoftDeleteMixin(object):
deleted_at = Column(DateTime)
deleted = Column(Integer, default=0)

def soft_delete(self, session=None):
"""Mark this object as deleted."""
self.deleted = self.id
self.deleted_at = timeutils.utcnow()
self.save(session=session)

Once mixed in to a concrete model class, the primary join is typically 
modified to include the deleted column, like so:


class ComputeNode(BASE, NovaBase):
...
service = relationship(Service,
   backref=backref('compute_node'),
   foreign_keys=service_id,
   primaryjoin='and_('
'ComputeNode.service_id == Service.id,'
'ComputeNode.deleted == 0)')

My proposal is to get rid of the deleted column in the SoftDeleteMixin 
class entirely, as it is redundant with the deleted_at column. Instead 
of doing a join condition on deleted == 0, one would instead just do the 
join condition on deleted_at is None, which translates to the SQL: AND 
deleted_at IS NULL.


There isn't much of a performance benefit -- you're only reducing the 
row size by 4 bytes. But, you'd remove the redundant data from all the 
tables, which would make the normal form freaks like myself happy ;)


Thoughts?

-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Proposal oslo.db lib

[Gary Kotton]

Hi,
I have a number of things to say here:

1.   Great work in getting the DB into the common and ironing out the issues

2.   As far as I know only Neutron and Nova are making use of the common DB 
code. Neutron has been using this since the beginning of H2 (this did not 
resolve all of the issues that we had and we had) and Nova has just recently 
upgraded to the latest DB code (this was a few weeks ago).

3.In general I like the idea of having a separate lib for this but have 
a number of reservations regarding the timing and stability:

a.   I do not think that this has been running long enough in Neutron and 
Nova for us to give it a stamp of approval (the common CFG code was at least 
one cycle as common code prior to moving into its own lib). I think that in 
Neutron we still have a number of issues with load on the DB. I need to double 
check on this.

b.  I think that the beginning of Icehouse is a good time. When we moved to 
the CFG library there were a number of hickups and issues along the way. I 
think that Mark (oslo PTL) can elaborate a little more on this. Timing is 
essential.

c.   In addition to this the migration from Neutron using it own DB 
implementation to the common required changes in the CFG library (we needed 
backward compatibility regarding configurations). Not sure if this may be the 
same for other projects migrating to the common code.
In short I'd like to see this wait till the beginning of Icehouse and would 
strongly suggest that we have a session on this at summit - it will certainly 
affect all of the projects using the common code.
Thanks
Gary

From: Joe Gordon [mailto:joe.gord...@gmail.com]
Sent: Monday, August 19, 2013 5:53 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] Proposal oslo.db lib



On Fri, Aug 16, 2013 at 11:36 AM, Monty Taylor 
mailto:mord...@inaugust.com>> wrote:


On 08/16/2013 09:52 AM, Boris Pavlovic wrote:
> Hi all,
>
> We (OpenStack contributors) done a really huge and great work around DB
> code in Grizzly and Havana to unify it, put all common parts into
> oslo-incubator, fix bugs, improve handling of sqla exceptions, provide
> unique keys, and to use  this code in different projects instead of
> custom implementations. (well done!)
>
> oslo-incubator db code is already used by: Nova, Neutron, Cinder,
> Ironic, Ceilometer.
>
> In this moment we finished work around Glance:
> https://review.openstack.org/#/c/36207/
>
> And working around Heat and Keystone.
>
> So almost all projects use this code (or planing to use it)
>
> Probably it is the right time to start work around moving oslo.db code
> to separated lib.
>
> We (Roman, Viktor and me) will be glad to help to make oslo.db lib:
>
> E.g. Here are two drafts:
> 1) oslo.db lib code: https://github.com/malor/oslo.db
> 2) And here is this lib in action:
> https://review.openstack.org/#/c/42159/
> 

Based on the discussion in the following thread, 
http://lists.openstack.org/pipermail/openstack-dev/2013-August/013746.html, and 
the amount of work listed as TODO there, I am concerned about the maturity of 
the API that oslo.db would produce, I think db may need to incubate longer in 
order to produce a stable API.

FYI: https://github.com/openstack/oslo-incubator/blob/master/MAINTAINERS shows 
db is orphaned ... which doesn't sound very accurate.


+1

Great job Boris!

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Jay Pipes]

On 08/18/2013 10:33 PM, Joe Gordon wrote:

An alternative I think would be better would be to scrap the use of
the SQLAlchemy ORM; keep using the DB engine abstraction support.

+1, I am hoping this will provide noticeable performance benefits while
being agnostic of what DB back-end is being used.  With the way we use
  SQLALchemy being 25x slower then MySQL we have lots of room for
improvement (see http://paste.openstack.org/show/44143/ from
https://bugs.launchpad.net/nova/+bug/1212418).


@require_admin_context
def compute_node_get_all(context):
return model_query(context, models.ComputeNode).\
options(joinedload('service')).\
options(joinedload('stats')).\
all()

Well, yeah... I suppose if you are attempting to create 115K objects in 
memory in Python (Need to collate each ComputeNode model object and each 
of its relation objects for Service and Stats) you are going to run into 
some performance problems. :)


Would be interesting to see what the performance difference would be if 
you instead had dicts instead of model objects and did something like 
this instead (code not tested, just off top of head...):


# Assume a method to_dict() that takes a Model
# and returns a dict with appropriate empty dicts for
# relationship fields.

qr = session.query(ComputeNode).join(Service).join(Stats)

results = {}

for record in qr:
  node_id = record.ComputeNode.id
  service_id = record.Service.id
  stat_id = record.ComputeNodeStat.id
  if node_id not in results.keys():
results[node_id] = to_dict(record.ComputeNode)
  if service_id not in results[node_id]['services'].keys():
results[node_id]['services'][service_id] = to_dict(record.Service)
  if stat_id not in results[node_id]['stats'].keys():
results[node_id]['stats'][stat_id] = to_dict(record.ComputeNodeStat)

return results

Whether it would be any faster than SQLAlchemy's joinedload...

Besides that, though, probably is a good idea to look at even the 
existence of DB calls that potentially do that kind of massive query 
returning as A Bad Thing...


Best,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

[Robert Collins]

On 19 August 2013 18:35, Jay Pipes  wrote:

>> http://www.codinghorror.com/blog/2006/06/object-relational-mapping-is-the-vietnam-of-computer-science.html
>>
>> There is no proper use of an ORM.
>
>
> I'm not a super-fan of ORMs, Robert. I'm not sure why you're insisting on
> taking me down this road...

Sorry, not sure how we ended up here ;)

> All I'm saying is that we should be careful not to swap one set of problems
> for another. I say this because I've seen the Nova data-access code develop
> from its very earliest days, up to this point. I've seen the horrors of
> trying to mask an object approach on top of a non-relational data store,
> witnessed numerous attempts to rewrite the way that connection pooling and
> session handling is done, and in general just noticed the tension between
> the two engineering factions that want to keep things agnostic towards
> backend storage and at the same time make the backend storage perform and
> scale adequately.

Ah! Ok, completely agree: playing flip-flop on problem sets would be a
poor outcome.

> I'm not sure why you are being so aggressive about this topic. I certainly
> am not being aggressive about my responses -- just cautioning that the
> existing codebase has seen its fair share of refactoring, some of which has
> been a failure and had to be reverted. I would hate to jump into a frenzy to
> radically change the way that the data access code works in Nova without a
> good discussion.

I didn't intend to be aggressive - sorry - super sorry in fact. I've
been burnt by months of effort turning around problem codebases where
the ORM was a significant cause of the problems.


>>> But then I guarantee somebody is gonna spend a bunch of time writing an
>>> object-oriented API to the model objects because the ORM is very useful
>>> for
>>> the data modification part of the DB interaction.
>>
>>
>> !cite - seriously...
>
>
> ? I give an example below... a cautionary tale if you will, about one
> possible consequence of "getting rid of the ORM".

I think what I really meant here is 'you say months, but if we're
writing an object-orientated API surely we'd just use one of the
mapping techniques available in SQLAlchemy..'

>> This strawman is one way that it might be written. Given that a
>> growing set of our projects have non-SQL backends, this doesn't look
>> like the obvious way to phrase it to me.
>
>
> I'm using the SQLAlchemy Core API above, with none of the SQLAlchemy ORM
> code... which is (I thought), what you were proposing we do? How is that a
> strawman argument? :(

So what is in my head is that we have two layers:
business logic
storage logic

And the thing I don't like about the ORM approach is that our business
logic objects are storage logic objects - even though we don't use
http://martinfowler.com/eaaCatalog/domainModel.html we can easily
trigger late evaluation when traversing collections. In particular
because we have large numbers of developers who are likely going to
not be holding the entire problem domain in their head; the churn that
results on code and design tends to throw things out again and again
over time. And we have IMO too much business logic in the
db/sqlalchemy/api.py files scattered around.

So, what I'd like to see is something where the storage layer and
logic layer are more thoroughly decoupled: only return plain ol Python
objects from the DB layer; but within that layer I wouldn't object to
an ORM being used; secondly I'd like to make sure we don't end up
making business decisions in the storage layer, because that makes it
harder when porting to a different storage layer - such as the nova
conductor is.

So the business logic layer for adding a fixed IP would be something like:
i = business.Instance.find(blah=blah)
ip = business.FixedIP(blah=blah)
i.fixed_ips.append(ip)
storage.save(i)

i and ip would be plain ol python objects
storage.save would have the same semantics as an RPC call - it could
do a transaction itself, but there's no holding transactions between
calls to save.

This is very close to:

>>> instead of this:
>>>
>>> i = Instance(blah=blah)
>>> ip = FixedIp(blah=blah)
>>> i.fixed_ips.append(ip)
>>> session.add(u)
>>> session.commit()

But there is no ORM exposed to the developers working with the storage
API - it's contained.

>>> And so you've thrown the baby out with the bathwater and made more work
>>> for
>>> everyone.
>>
>>
>> Perhaps; or perhaps we've avoided a raft of death-by-thousand-cuts
>> bugs across the project.
>
>
> Could just as easily introduce the same bugs by radically redesigning the
> data access code without first considering all sides of the problem domain.

Totally!

Again, sorry for the tone before, I can only claim a) been burnt in
the past and and b) a week or so of reduced sleep thanks to baby :(.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
h