Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Mark McLoughlin
On Fri, 2014-08-08 at 09:06 -0400, Russell Bryant wrote:
 On 08/07/2014 08:06 PM, Michael Still wrote:
  It seems to me that the tension here is that there are groups who
  would really like to use features in newer libvirts that we don't CI
  on in the gate. Is it naive to think that a possible solution here is
  to do the following:
  
   - revert the libvirt version_cap flag
 
 I don't feel strongly either way on this.  It seemed useful at the time
 for being able to decouple upgrading libvirt and enabling features that
 come with that.

Right, I suggested the flag as a more deliberate way of avoiding the
issue that was previously seen in the gate with live snapshots. I still
think it's a pretty elegant and useful little feature, and don't think
we need to use it as proxy battle over testing requirements for new
libvirt features.

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Python3 search path order when building Debian package

2014-08-11 Thread Thomas Goirand
Hi,

During the build of oslo packages for OpenStack, I'm having the issue
that python3 seems to first look into
/usr/lib/python3.4/dist-packages/olso, finds stuff in it, and then do
not search in the local directory. Because of that, unit tests are
failing during the build. For example, when trying to build
python-oslo.utils (new package, not yet available in Debian), I get:

==
FAIL: unittest.loader.ModuleImportFailure.tests.test_excutils
unittest.loader.ModuleImportFailure.tests.test_excutils
--
_StringException: Traceback (most recent call last):
ImportError: Failed to import test module: tests.test_excutils
Traceback (most recent call last):
  File /usr/lib/python3.4/unittest/loader.py, line 312, in _find_tests
module = self._get_module_from_name(name)
  File /usr/lib/python3.4/unittest/loader.py, line 290, in
_get_module_from_name
__import__(name)
  File
/home/zigo/sources/openstack/juno/python-oslo.utils/build-area/python-oslo.utils-0.1.1/tests/test_excutils.py,
line 22, in module
from oslo.utils import excutils
ImportError: No module named 'oslo.utils'

note that I am running testr the following way:

testr-python3 init
PYTHON=python3.4 testr-python3 run --subunit | subunit2pyunit

Of course, if I first install python3-oslo.utils, then the unit tests
are passing without any issue. Though at build time, the package isn't
installed in the system yet.

What's very surprising to me, is that I have this issue only with Python
3, not with python 2.7, which passes the tests without any issue, and
does the path hunt for the python module correctly.

For other oslo libraries, I have declared some Build-Conflicts, to make
sure that they aren't installed in the system when building. Though that
can't be done with oslo.utils, because it depends on oslo.config and
oslo.i18n.

I think this highlight once more that using namespace should be avoided
if possible. This creates more issues than it solves problems.

Any idea, comment or whatever that may help would be appreciated.

Cheers,

Thomas Goirand (zigo)

P.S: By the way, there's issues with python-oslo.utils with Python 3.4,
after solving the above problem...

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][LBaaS] Improvements to current reviews

2014-08-11 Thread Susanne Balle
I agree with Doug as well. We should update the current patch. Susanne


On Sun, Aug 10, 2014 at 8:18 AM, Gary Kotton gkot...@vmware.com wrote:

  Hi,
 I took a look at https://review.openstack.org/#/c/105331/ and had one
 minor issue which I think can be addressed. Prior to approving we need to
 understand what the state of the V2 API will be.
 Thanks
 Gary

   From: Vijay Venkatachalam vijay.venkatacha...@citrix.com
 Reply-To: OpenStack List openstack-dev@lists.openstack.org
 Date: Sunday, August 10, 2014 at 2:57 PM
 To: OpenStack List openstack-dev@lists.openstack.org

 Subject: Re: [openstack-dev] [Neutron][LBaaS] Improvements to current
 reviews

   Thanks Brandon for constant  improvisation.

 I agree with Doug. Please update current one. We already hv  more number
 of reviews :-). It will be difficult to manage if we add more.

 Thanks,
 Vijay

 Sent using CloudMagic

 On Sun, Aug 10, 2014 at 3:23 AM, Doug Wiegley do...@a10networks.com
 wrote:

  I think you should update the current reviews (new patch set, not
 additional review.)

 Doug


  On Aug 9, 2014, at 3:34 PM, Brandon Logan brandon.lo...@rackspace.com
 wrote:
 
  So I've done some work on improving the code on the current pending
  reviews.  And would like to get peoples' opinions on whether I should
  add antoher patch set to those reviews, or add the changes as as another
  review dependent on the pending ones.
 
  To be clear, no matter what the first review in the chain will not
  change:
  https://review.openstack.org/#/c/105331/
 
  However, if adding another patch set is preferrable then plugin and db
  implementation review would get another patch set and then obviously
  anything depending on that.
 
  https://review.openstack.org/#/c/105609/
 
  My opinion is that I'd like to get both of these in as a new patch set.
  Reason being that the reviews don't have any +2's and there is
  uncertainty because of the GBP discussion.  So, I don't think it'd be a
  major issue if a new patch set was created.  Speak up if you think
  otherwise.  I'd like to get as many people's thoughts as possible.
 
  The changes are:
 
  1) Added data models, which are just plain python object mimicking the
  sql alchemy models, but don't have the overhead or dynamic nature of
  being sql alchemy models.  These data models are now returned by the
  database methods, instead of the sql alchemy objects.  Also, I moved the
  definition of the sql alchemy models into its own module.  I've been
  wanting to do this but since I thought I was running out of time I left
  it for later.
 
  These shouldn't cause many merge/rebase conflicts, but it probably cause
  a few because the sql alchemy models were moved to a different module.
 
 
  2) The LoadBalancerPluginv2 no longer inherits from the
  LoadBalancerPluginDbv2.  The database is now a composite attribute of
  the plugin (i.e. plugin.db.get_loadbalancer()).  This cleans up the code
  a bit and removes the necessity for _delete_db_entity methods that the
  drivers needed because now they can actually call the database methods.
  Also, this makes testing more clear, though I have not added any tests
  for this because the previous tests are sufficient for now.  Adding the
  appropriate tests would add 1k or 2k lines most likely.
 
  This will likely cause more conflicts because the _db_delete_entity
  methods have been removed.  However, the new driver interface/mixins
  defined a db_method for all drivers to use, so if that is being used
  there shouldn't be any problems.
 
  Thanks,
  Brandon
 
 
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: FW: [Neutron] Group Based Policy and the way forward

2014-08-11 Thread loy wolfe
Hi Sumit,

First I want to say I'm not opposed to GBP itself, but has many confusion
about it's core resource model and how it will integrate with neutron core.

Do you mean for whatever GBP backend configured in any future Neutron
deployment, so long as they are in tree, then ML2 core plugin shall always
be there to expose the Neutron core resource: Network/Subnet/Port?

Then since Network/Subnet/Port will never be treated just as LEGACY
COMPATIBLE role, there is no need to extend Nova-Neutron interface to
follow the GBP resource. Anyway, one of optional service plugins inside
Neutron shouldn't has any impact on Nova side.

If we agree on this point, core model of GBP should be reviewed, but not
just focus on naming convention whether it should be called EP or policy
target, then leaving some words here to emphasize GBP is only
complementary. In fact, EP/EPG/BD/RD has been designed to be ABLE TO
REPLACE Neutron core resource, with mapping as the first step to keep
compatible.

In fact, if Neutron core source shall never to be swapped out, GBP core
object could be greatly simplified, because mapping already means
redundancy :-) Only policy-group is meaningful, behind which are very
important policy concept: consumer/producer contracts. After PG is defined,
it should be directly applied to existing Neutron core resource, but not
create similar concept of EP/L2P/L3P, then mapping to them. Mapping is
redundant, and I can understand it's necessity only if someday those
neutron core resource are planned to be swapped out.

Simple conclusion: if GBP is just an optional complementary, then after
defining policy template, directly APPLYING, but not create similar
redundant resource and then MAPPING it to existing neutron core resource.


On Sat, Aug 9, 2014 at 3:35 AM, Jay Pipes jaypi...@gmail.com wrote:

 On 08/08/2014 12:29 PM, Sumit Naiksatam wrote:

 Hi Jay, To extend Ivar's response here, the core resources and core
 plugin configuration does not change with the addition of these
 extensions. The mechanism to implement the GBP extensions is via a
 service plugin. So even in a deployment where a GBP service plugin is
 deployed with a driver which interfaces with a backend that perhaps
 directly understands some of the GBP constructs, that system would
 still need to have a core plugin configured that honors Neutron's core
 resources. Hence my earlier comment that GBP extensions are
 complementary to the existing core resources (in much the same way as
 the existing extensions in Neutron).


 OK, thanks Sumit. That clearly explains things for me.

 Best,
 -jay



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Octavia] Minutes from 8/6/2014 meeting

2014-08-11 Thread Susanne Balle
Great notes! thanks it helped me catch up after vacation. :)


On Thu, Aug 7, 2014 at 4:33 AM, Stephen Balukoff sbaluk...@bluebox.net
wrote:

 On where to capture notes like this long-term:  I would say the wiki is
 more searchable for now. When we make the transition to IRC meetings, then
 the meeting bots will capture minutes and transcripts in the usual way and
 we can link to these from the wiki.


 On Thu, Aug 7, 2014 at 1:29 AM, Stephen Balukoff sbaluk...@bluebox.net
 wrote:

 Wow, Trevor! Thanks for capturing all that!


 On Wed, Aug 6, 2014 at 9:47 PM, Trevor Vardeman 
 trevor.varde...@rackspace.com wrote:

 Agenda items are numbered, and topics, as discussed, are described
 beneath in list format.

 1) Octavia Constitution and Project Direction Documents (Road map)
 a) Constitution and Road map will potentially be adopted after
 another couple days; providing those who were busy more time to review the
 information

 2) Octavia Design Proposals
 a) Difference between version 0.5 and 1.0 isn't huge
 b) Version 2 has many network topology changes and Layer 4 routing
 + This includes N node Active-Active
 + Would like to avoid Layer 2 connectivity with Load Balancers
 (included in version 1 however)
 + Layer router driver
 + Layer router controller
 + Long term solution
 c) After refining Version 1 document (with some scrutiny) all
 changes will be propagated to the Version 2 document
 d) Version 0.5 is unpublished
 e) All control layer, anything connected to the intermediate message
 bus in version 1, will be collapsed down to 1 daemon.
 + No scale-able control, but scale-able service delivery
 + Version 1 will be the first large operator compatible version,
 that will have both scale-able control and scale-able service delivery
 + 0.5 will be a good start
 - laying out ground work
 - rough topology for the end users
 - must be approved by the networking teams for each
 contributing company
 f) The portions under control of neutron lbaas is the User API and
 the driver (for neutron lbaas)
 g) If neutron LBaaS is a sufficient front-end (user API doesn't
 suck), then Octavia will be kept as a vendor driver
 h) Potentially including a REST API on top of Octavia
 + Octavia is initially just a vendor driver, no real desire for
 another API in front of Octavia
 + If someone wants it, the work is trivial and can be done in
 another project at another time
 i) Octavia should have a loose coupling with Neutron; use a shim for
 network connectivity (one specifically for Neutron communication in the
 start)
 + This is going to hold any dirty hacks that would be required
 to get something done, keeping Octavia clean
 - Example: changing the mac address on a port

 3) Operator Network Topology Requirements
 a) One requirement is floating IPs.
 b) IPv6 is in demand, but is currently not supported reliably on
 Neutron
 + IPv6 would be represented as a different load balancer entity,
 and possibly include co-location with another Load Balancer
 c) Network interface plug-ability (potentially)
 d) Sections concerning front-end connectivity should be forwarded to
 each company's network specialists for review
 + Share findings in the mailing list, and dissect the proposals
 with the information and comment what requirements are needing added etc.

 4) HA/Failover Options/Solutions
 a) Rackspace may have a solution to this, but the conversation will
 be pushed off to the next meeting (at least)
 + Will gather more information from another member in Rackspace
 to provide to the ML for initial discussions
 b) One option for HA:  Spare pool option (similar to Libra)
 + Poor recovery time is a big problem
 c) Another option for HA:  Active/Passive
 + Bluebox uses one active and one passive configuration, and has
 sub-second fail over.  However is not resource-sufficient

 Questions:
 Q:  What is the expectation for a release time-frame
 A:  Wishful thinking; Octavia version 0.5 beta for Juno (probably not,
 but would be awesome to push for that)

 Notes:
  + We need to pressure the Neutron core reviewers to review the Neutron
 LBaaS changes to get merges.
  + Version 2 front-end topology is different than the Version 1.  Please
 review them individually, and thoroughly


 PS.  I re-wrote most of the information from the recording (thanks again
 Doug).  I have one question for everyone: should I just email this out
 after each meeting to the Octavia mailing list, or should I also add it to
 a page in an Octavia wiki for Meeting Notes/Minutes or something for review
 by anyone?  What are your thoughts?

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 

[openstack-dev] [Heat] SoftwareDeployment resource is always in progress

2014-08-11 Thread david ferahi
Hello,

I 'm trying to create a simple stack with heat (Icehouse release).
The template contains SoftwareConfig, SoftwareDeployment and a single
server resources.

The problem is that the SoftwareDeployment resource is always in progress !

After waiting for more than an hour the stack deployment failed and I got
this error:

 TRACE heat.engine.resource HTTPUnauthorized: ERROR: Authentication failed.
Please try again with option --include-password or export
HEAT_INCLUDE_PASSWORD=1
TRACE heat.engine.resource Authentication required

When I checked the log file (/var/log/heat/heat-engine.log), it shows  the
following message(every second):
2014-08-10 19:41:09.622 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10
2014-08-10 19:41:10.648 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10
2014-08-10 19:41:11.671 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10
2014-08-10 19:41:12.690 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10

Here the template I am using :
https://github.com/openstack/heat-templates/blob/master/hot/software-config/example-templates/wordpress/WordPress_software-config_1-instance.yaml

Please help !

Kind Regards,

David
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree

2014-08-11 Thread Thierry Carrez
gustavo panizzo (gfa) wrote:
 only one think i didn't like it
 
 why all url,api, etc has to include the word 'preview'?
 i imagine that i would be consuming the new feature using heat, puppet,
 local scripts, custom horizon, whatever. Why do you make me to change
 all them when the feature moves out of preview? it could be a lot of
 rework (for consumers) without gain. I would totally support other big
 fat warnings everywhere (logs, documentation, startup log of
 neutron-server) but don't change the API if is not necessary

I see two issues with this proposal: the first one is what Gustavo just
said: the use of the preview package/configoption/API creates friction
when the feature needs to go mainstream (package renaming, change in
configuration for deployers, change in API calls for users...).

My understanding is that the goal is to make it easy for people to try
the preview feature, and keeping the experimental feature in-tree is
seen as simpler to experiment with. But the pain from this friction imho
outweighs the pain of deploying an out-of-tree plugin for deployers.

The second issue is that once the feature is in preview in tree, it
moves the responsibility/burden of making it official (or removed) to
the core developers (as Salvatore mentioned). I kind of like the
approach where experimental features are developed in faster iterations
out-of-tree and when they are celebrated by experimenters and are ready
to be stable-supported, they are moved in tree.

Regards,

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [keystone] how to generate conf sample in havana?

2014-08-11 Thread Li Tianqing
Hello,
   Right now, i customerized keystone to meet our requeriments. I want to add 
some configuration options in keystone.conf.sample. But, i 
do not find generate_sample.sh in directory tools in Havana keystone. So i want 
to know how to generate conf sample in havana keystone.
I know, i can write directly to keystone.conf.sample, but should there be 
something like generate_sample.sh in keystone?___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] The future of the integrated release

2014-08-11 Thread Thierry Carrez
Jeremy Stanley wrote:
 On 2014-08-08 09:43:54 -0700 (-0700), Devananda van der Veen wrote:
 [...]
 this sounds like it will also solve the current problem when a
 core reviewer has gone on vacation after blocking something for
 procedural reasons.
 
 I don't really think so, unless I'm misunderstanding which vacation
 problem you're talking about. We can either make a vote carry over
 on subsequent patchsets (in which case you need someone to reverse
 or delete the -2 before you can merge it) or not (in which case you
 will lose that -2 on the next uploaded patchset and have to keep
 reapplying it manually).
 
 Unless perhaps you're suggesting that we should allow approved
 changes to merge even with a workflow -2 from another reviewer
 (keeping in mind that we don't currently allow changes with a
 workflow -1 to merge).

Hmm..

Ideally we'd want a blocking vote that carries over patchsets, but which
can be set or unset by any core/driver/ptl (to be determined). So if the
PTL sets the procedural Workflow-2 at feature freeze, I can clear it
when RC1 is published.

Not sure that's supported in Gerrit though :)

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Neutron][LBaaS] Use cases with regards to VIP and routers

2014-08-11 Thread Susanne Balle
Gang,

I was asked the following questions around our Neutron LBaaS use cases:

1.  Will there be a scenario where the “VIP” port will be in a different
Node, from all the Member “VMs” in a pool.

2.  Also how likely is it for the LBaaS configured subnet to not have a
“router” and just use the “extra_routes” option.

3.  Is there a valid use case where customers will be using the
“extra_routes” with subnets instead of the “routers”. ( It would be great
if you have some use case picture for this).

Feel free to chime in here and I'll summaries the answers.

Regards Susanne
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] Using host networking for docker containers

2014-08-11 Thread Aleksandr Didenko
Hi,

we're running only 3 containers in privileged mode: cobbler, rsyslog and
mcollective. Running all the containers in privileged mode is not a good
idea for security reasons. Docker manages DNAT forwarding itself, so it
does not create any overhead for us.

 Is there any real benefits of using separate namespaces in security terms?

Of course, for example only ports specified in EXPOSE line in Dockerfile
are exposed to the host network. So if you start any additional tcp/udp
listeners inside the containers, their ports won't be accessible from the
host network.



On Sat, Aug 9, 2014 at 10:39 AM, Dmitriy Shulyak dshul...@mirantis.com
wrote:

 Hi team,

 I want to discuss benefits of using host networking [1] for docker
 containers, on master node.

 This feature was added in docker 0.11 and basicly means - reuse host
 networking stack, without
 creating separate namespace for each container.

 In my opinion it will result in much more stable install/upgrade of master
 node.

 1. There will be no need for dhcrelay/dhcrelay_monitor on host
 2. No dnat port forwarding
 3. Performance improvement for pxe boot ???

 Is there any real benefits of using separate namespaces in security terms?

 To implement this we will need:

 1. Update docker to recent version 0.12/1.x, we will do it anyway, yes?
 2. Run docker containers with --net=host

 Ofcourse it will require running containers in privileged mode, but afaik
 we are already doing this for other reasons.

 So, what do you think?

 [1] https://github.com/docker/docker/issues/2012
 [2] https://docs.docker.com/articles/networking/

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] The future of the integrated release

2014-08-11 Thread Thierry Carrez
Eoghan Glynn wrote:
 The TC has begun to scrutinize new projects more carefully on
 technical grounds, particularly since some recently-integrated
 projects have run into scaling limitations that have hampered their
 adoption. I believe this sort of technical guidance (or curation, if
 you will) is an essential function of the TC. We've learned that
 integration bestows legitimacy, as well as assumptions of stability,
 performance, and scalability, upon projects which are integrated.
 While that wasn't the intent, I think we need to accept that that is
 where we stand. We will be faced with some hard decisions regarding
 projects, both incubated and already integrated, which are clearly not
 meeting those expectations today.
 
 How does this relate to the recent gap analysis undertaken by the
 TC for already integrated projects, in order to measure their status
 against the steadily rising bar for integration?

It is almost orthogonal. During the Icehouse cycle the TC created clear
requirements for projects wishing to incubate or graduate for
incubation. But it was unclear how well the *currently-incubated*
projects fulfilled those requirements. Since it was a bit unfair to hold
new entrants to different standards, during the Juno cycle we completed
a gap analysis of currently-integrated projects, created gap coverage
plans to address any identified gap, and reviewed progress on those
after each milestone.

So the idea that being (and remaining) in the integrated release should
also be judged on technical merit is a slightly different effort. It's
always been a factor in our choices, but like Devananda says, it's more
difficult than just checking a number of QA/integration checkboxes. In
some cases, blessing one project in a problem space stifles competition,
innovation and alternate approaches. In some other cases, we reinvent
domain-specific solutions rather than standing on the shoulders of
domain-specific giants in neighboring open source projects.

This all has created a world where you need to be *in* OpenStack to
matter, or to justify the investment. This has created a world where
everything and everyone wants to be in the OpenStack integrated
release. This has created more pressure to add new projects, and less
pressure to fix and make the existing projects perfect. 4 years in, we
might want to inflect that trajectory and take steps to fix this world.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

2014-08-11 Thread Daniel P. Berrange
On Mon, Aug 04, 2014 at 06:46:13PM -0400, Solly Ross wrote:
 Hi,
 I was wondering if there was a way to get a non-readonly connection
 to libvirt when running the unit tests
 on the CI.  If I call `LibvirtDriver._connect(LibvirtDriver.uri())`,
 it works fine locally, but the upstream
 CI barfs with libvirtError: internal error Unable to locate libvirtd
 daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the
 libvirtd binary).
 If I try to connect by calling libvirt.open(None), it also barfs, saying
 I don't have permission to connect.  I could just set it to always use
 fakelibvirt,
 but it would be nice to be able to run some of the tests against a real
 target.  The tests in question are part of 
 https://review.openstack.org/#/c/111459/,
 and involve manipulating directory-based libvirt storage pools.

Nothing in the unit tests should rely on being able to connect to the
libvirt daemon, as the unit tests should still be able to pass when the
daemon is not running at all. We should be either using fakelibvirt or
mocking any libvirt APIs that need to be tested

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Daniel P. Berrange
On Mon, Aug 11, 2014 at 08:57:31AM +1000, Michael Still wrote:
 On Sun, Aug 10, 2014 at 2:33 AM, Jeremy Stanley fu...@yuggoth.org wrote:
  On 2014-08-08 09:06:29 -0400 (-0400), Russell Bryant wrote:
  [...]
  We've seen several times that building and maintaining 3rd party
  CI is a *lot* of work.
 
  Building and maintaining *any* CI is a *lot* of work, not the least
  of which is the official OpenStack project CI (I believe Monty
  mentioned in #openstack-infra last night that our CI is about twice
  the size of Travis-CI now, not sure what metric he's comparing there
  though).
 
  Like you said in [1], doing this in infra's CI would be ideal. I
  think 3rd party should be reserved for when running it in the
  project's infrastructure is not an option for some reason
  (requires proprietary hw or sw, for example).
 
  Add to the not an option for some reason list, software which is
  not easily obtainable through typical installation channels (PyPI,
  Linux distro-managed package repositories for their LTS/server
  releases, et cetera) or which requires gyrations which destabilize
  or significantly complicate maintenance of the overall system as
  well as reproducibility for developers. It may be possible to work
  around some of these concerns via access from multiple locations
  coupled with heavy caching, but adding that in for a one-off source
  is hard to justify the additional complexity too.
 
 My understanding is that Fedora has a PPA equivalent which ships a
 latest and greated libvirt. So, it would be packages if we went the
 Fedora route, which should be less work.

Yes, there is the 'virt preview' repository

http://fedoraproject.org/wiki/Virtualization_Preview_Repository

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Daniel P. Berrange
On Fri, Aug 08, 2014 at 09:06:29AM -0400, Russell Bryant wrote:
 On 08/07/2014 08:06 PM, Michael Still wrote:
  It seems to me that the tension here is that there are groups who
  would really like to use features in newer libvirts that we don't CI
  on in the gate. Is it naive to think that a possible solution here is
  to do the following:
  
   - revert the libvirt version_cap flag
 
 I don't feel strongly either way on this.  It seemed useful at the time
 for being able to decouple upgrading libvirt and enabling features that
 come with that.  I'd like to let Dan get back from vacation and weigh in
 on it, though.

Yes, I think that version cap feature is valuable no matter what we
do about CI testing, which is why I +2'd it originally.

 
 I wonder if the job could be as simple as one with an added step in the
 config to install latest libvirt from source.  Dan, do you think someone
 could add a libvirt-current.tar.gz to http://libvirt.org/sources/ ?
 Using the latest release seems better than master from git.

I'd strongly recommend against using GIT master. It will cause openstack
CI more pain than benefits. Using the latest stable release is a better
bet

 I'll mess around and see if I can spin up an experimental job.

There is work to add support for this in devestack already which I
prefer since it makes it easy for developers to get an environment
which matches the build system:

  https://review.openstack.org/#/c/108714/

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] Using host networking for docker containers

2014-08-11 Thread Matthew Mosesohn
Moving to host networking would reduce our ability to do zero downtime
upgrades in the future. It means you must kill the old container in
order to start the new one, rather than allowing for the possibility
to remap the network configuration in iptables. It's something we
don't have now, but we may be able to do in the future.

With regards to security issues, we have some more restrictive
firewall rules in place now. I don't think this is a major issue.

I don't think it makes a huge difference in performance to switch to
host networking, but it's worth testing.

On Mon, Aug 11, 2014 at 1:16 PM, Aleksandr Didenko
adide...@mirantis.com wrote:
 Hi,

 we're running only 3 containers in privileged mode: cobbler, rsyslog and
 mcollective. Running all the containers in privileged mode is not a good
 idea for security reasons. Docker manages DNAT forwarding itself, so it does
 not create any overhead for us.


 Is there any real benefits of using separate namespaces in security terms?

 Of course, for example only ports specified in EXPOSE line in Dockerfile are
 exposed to the host network. So if you start any additional tcp/udp
 listeners inside the containers, their ports won't be accessible from the
 host network.



 On Sat, Aug 9, 2014 at 10:39 AM, Dmitriy Shulyak dshul...@mirantis.com
 wrote:

 Hi team,

 I want to discuss benefits of using host networking [1] for docker
 containers, on master node.

 This feature was added in docker 0.11 and basicly means - reuse host
 networking stack, without
 creating separate namespace for each container.

 In my opinion it will result in much more stable install/upgrade of master
 node.

 1. There will be no need for dhcrelay/dhcrelay_monitor on host
 2. No dnat port forwarding
 3. Performance improvement for pxe boot ???

 Is there any real benefits of using separate namespaces in security terms?

 To implement this we will need:

 1. Update docker to recent version 0.12/1.x, we will do it anyway, yes?
 2. Run docker containers with --net=host

 Ofcourse it will require running containers in privileged mode, but afaik
 we are already doing this for other reasons.

 So, what do you think?

 [1] https://github.com/docker/docker/issues/2012
 [2] https://docs.docker.com/articles/networking/

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Gary Kotton
In the past there was smokestack that used a fedora version of libvirt.
Any idea what happened to that?

On 8/11/14, 12:53 PM, Daniel P. Berrange berra...@redhat.com wrote:

On Fri, Aug 08, 2014 at 09:06:29AM -0400, Russell Bryant wrote:
 On 08/07/2014 08:06 PM, Michael Still wrote:
  It seems to me that the tension here is that there are groups who
  would really like to use features in newer libvirts that we don't CI
  on in the gate. Is it naive to think that a possible solution here is
  to do the following:
  
   - revert the libvirt version_cap flag
 
 I don't feel strongly either way on this.  It seemed useful at the time
 for being able to decouple upgrading libvirt and enabling features that
 come with that.  I'd like to let Dan get back from vacation and weigh in
 on it, though.

Yes, I think that version cap feature is valuable no matter what we
do about CI testing, which is why I +2'd it originally.

 
 I wonder if the job could be as simple as one with an added step in the
 config to install latest libvirt from source.  Dan, do you think someone
 could add a libvirt-current.tar.gz to
https://urldefense.proofpoint.com/v1/url?u=http://libvirt.org/sources/k=
oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPh
CZFxPEq8%3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=0298
f3506d0ac41e81fce6b0b3096455c3e78ba578970e42290997a766b0811e ?
 Using the latest release seems better than master from git.

I'd strongly recommend against using GIT master. It will cause openstack
CI more pain than benefits. Using the latest stable release is a better
bet

 I'll mess around and see if I can spin up an experimental job.

There is work to add support for this in devestack already which I
prefer since it makes it easy for developers to get an environment
which matches the build system:

  https://review.openstack.org/#/c/108714/

Regards,
Daniel
-- 
|: 
https://urldefense.proofpoint.com/v1/url?u=http://berrange.com/k=oIvRg1%2
BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%
3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=d7e4cac70435cd
1066f4f7ec8795f16c345c0ca4529bb4b3033762552817e775  -o-
https://urldefense.proofpoint.com/v1/url?u=http://www.flickr.com/photos/db
errange/k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfD
tysg45MkPhCZFxPEq8%3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%
0As=0f30863dcc9a34ffe520b0cee787f76bb67334963ebcd57d9b53b3890f0265ea :|
|: 
https://urldefense.proofpoint.com/v1/url?u=http://libvirt.org/k=oIvRg1%2B
dGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3
D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=07044ea24f9483f
b87f1c67511c4c62a3e2b8971ad8418b7a0a61068f623f529  -o-
 
https://urldefense.proofpoint.com/v1/url?u=http://virt-manager.org/k=oIvR
g1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxP
Eq8%3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=185d42ae1a
5b0422c1fa3306e55397461140024c38530f3a6986a72c51edec08 :|
|: 
https://urldefense.proofpoint.com/v1/url?u=http://autobuild.org/k=oIvRg1%
2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8
%3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=2e295d841588d
8cb243a6c37eb0c17007760ba0506cd621fff34de6a6f7c7b03   -o-
https://urldefense.proofpoint.com/v1/url?u=http://search.cpan.org/~danberr
/k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45M
kPhCZFxPEq8%3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=39
b473e1829f6e6c4f7bd4e27304334df8c0d1e9ac2cf7432490ac72fbe111e6 :|
|: 
https://urldefense.proofpoint.com/v1/url?u=http://entangle-photo.org/k=oI
vRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZF
xPEq8%3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=c100dc7d
e638a7bd0477a6d9723599041f442e80207213597a220e1a2cd34c74   -o-
https://urldefense.proofpoint.com/v1/url?u=http://live.gnome.org/gtk-vnck
=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0Ar=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPh
CZFxPEq8%3D%0Am=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0As=5c43d
0b2c8ad31d53813420ce9ab1885943b4ef88458006edcb2295f47c46846 :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Vijay Venkatachalam
Hi:

Continuing from last week's LBaaS meeting...

Currently an entity cannot be sent to driver unless it is linked to 
loadbalancer because loadbalancer is the root object and driver information is 
only available with loadbalancer.

The request to the driver is delayed because of which error propagation becomes 
tricky.

Let's say a monitor was configured with timeout  delay there would be no error 
then.
When a listener is configured there will be a monitor creation/deployment error 
like timeout configured greater than delay.

Unless the error is very clearly crafted the user won't be able to understand 
the error.

I am half-heartedly OK with current approach.

But, I would prefer Brandon's Solution - make provider an attribute in each of 
the entities to get rid of this problem.

What do others think?

Thanks,
Vijay V.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Russell Bryant
On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
 There is work to add support for this in devestack already which I
 prefer since it makes it easy for developers to get an environment
 which matches the build system:
 
   https://review.openstack.org/#/c/108714/

Ah, cool.  Devstack is indeed a better place to put the build scripting.
 So, I think we should:

1) Get the above patch working, and then merged.

2) Get an experimental job going to use the above while we work on #3

3) Before the job can move into the check queue and potentially become
voting, it needs to not rely on downloading the source on every run.
IIRC, we can have nodepool build an image to use for these jobs that
includes the bits already installed.

I'll switch my efforts over to helping get the above completed.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Daniel P. Berrange
On Mon, Aug 11, 2014 at 07:58:41AM -0400, Russell Bryant wrote:
 On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
  There is work to add support for this in devestack already which I
  prefer since it makes it easy for developers to get an environment
  which matches the build system:
  
https://review.openstack.org/#/c/108714/
 
 Ah, cool.  Devstack is indeed a better place to put the build scripting.
  So, I think we should:
 
 1) Get the above patch working, and then merged.
 
 2) Get an experimental job going to use the above while we work on #3
 
 3) Before the job can move into the check queue and potentially become
 voting, it needs to not rely on downloading the source on every run.

Don't we have the ability to mirror downloads locally to the build
system for python ?  The proposed patch allows an alternate download
URL to be set via an env variable so it could point to a local mirror
instead of libvirt.org / qemu.org

 IIRC, we can have nodepool build an image to use for these jobs that
 includes the bits already installed.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Russell Bryant
On 08/09/2014 12:33 PM, Jeremy Stanley wrote:
 On 2014-08-08 09:06:29 -0400 (-0400), Russell Bryant wrote:
 [...]
 We've seen several times that building and maintaining 3rd party
 CI is a *lot* of work.
 
 Building and maintaining *any* CI is a *lot* of work, not the least
 of which is the official OpenStack project CI (I believe Monty
 mentioned in #openstack-infra last night that our CI is about twice
 the size of Travis-CI now, not sure what metric he's comparing there
 though).

Dang, I'd love to see those numbers.  :-)

 Like you said in [1], doing this in infra's CI would be ideal. I
 think 3rd party should be reserved for when running it in the
 project's infrastructure is not an option for some reason
 (requires proprietary hw or sw, for example).
 
 Add to the not an option for some reason list, software which is
 not easily obtainable through typical installation channels (PyPI,
 Linux distro-managed package repositories for their LTS/server
 releases, et cetera) or which requires gyrations which destabilize
 or significantly complicate maintenance of the overall system as
 well as reproducibility for developers. It may be possible to work
 around some of these concerns via access from multiple locations
 coupled with heavy caching, but adding that in for a one-off source
 is hard to justify the additional complexity too.

Understood.  Some questions ... is building an image that has libvirt
and qemu pre-installed from source good enough?  It avoids the
dependency on job runs, but moves it to image build time though, so it
still exists.

If the above still doesn't seem like a workable setup, then I think we
should just go straight to an image with fedora + virt-preview repo,
which kind of sounds easier, anyway.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Russell Bryant
On 08/11/2014 07:58 AM, Russell Bryant wrote:
 On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
 There is work to add support for this in devestack already which I
 prefer since it makes it easy for developers to get an environment
 which matches the build system:

   https://review.openstack.org/#/c/108714/
 
 Ah, cool.  Devstack is indeed a better place to put the build scripting.
  So, I think we should:
 
 1) Get the above patch working, and then merged.
 
 2) Get an experimental job going to use the above while we work on #3
 
 3) Before the job can move into the check queue and potentially become
 voting, it needs to not rely on downloading the source on every run.
 IIRC, we can have nodepool build an image to use for these jobs that
 includes the bits already installed.
 
 I'll switch my efforts over to helping get the above completed.
 

I still think the devstack patch is good, but after some more thought, I
think a better long term CI job setup would just be a fedora image with
the virt-preview repo.  I think I'll try that ...

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Russell Bryant
On 08/11/2014 08:01 AM, Daniel P. Berrange wrote:
 On Mon, Aug 11, 2014 at 07:58:41AM -0400, Russell Bryant wrote:
 On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
 There is work to add support for this in devestack already which I
 prefer since it makes it easy for developers to get an environment
 which matches the build system:

   https://review.openstack.org/#/c/108714/

 Ah, cool.  Devstack is indeed a better place to put the build scripting.
  So, I think we should:

 1) Get the above patch working, and then merged.

 2) Get an experimental job going to use the above while we work on #3

 3) Before the job can move into the check queue and potentially become
 voting, it needs to not rely on downloading the source on every run.
 
 Don't we have the ability to mirror downloads locally to the build
 system for python ?  The proposed patch allows an alternate download
 URL to be set via an env variable so it could point to a local mirror
 instead of libvirt.org / qemu.org

There's a pypi mirror at least.  I'm not sure about mirroring other things.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] The future of the integrated release

2014-08-11 Thread Jeremy Stanley
On 2014-08-11 11:02:52 +0200 (+0200), Thierry Carrez wrote:
 Ideally we'd want a blocking vote that carries over patchsets, but which
 can be set or unset by any core/driver/ptl (to be determined). So if the
 PTL sets the procedural Workflow-2 at feature freeze, I can clear it
 when RC1 is published.
 
 Not sure that's supported in Gerrit though :)

As far as I know it's not... otherwise we'd have made WIP do the
same as you describe.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Nova] PCI support

2014-08-11 Thread Gary Kotton
Hi,
At the moment all of the drivers are required CI support. Are there any plans 
regarding the PIC support. I understand that this is something that requires 
specific hardware. Are there any plans to add this?
Thanks
Gary
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: FW: [Neutron] Group Based Policy and the way forward

2014-08-11 Thread CARVER, PAUL
loy wolfe [mailto:loywo...@gmail.com] wrote:

Then since Network/Subnet/Port will never be treated just as LEGACY
COMPATIBLE role, there is no need to extend Nova-Neutron interface to
follow the GBP resource. Anyway, one of optional service plugins inside
Neutron shouldn't has any impact on Nova side.

This gets to the root of why I was getting confused about Jay and others
having Nova related concerns. I was/am assuming that GBP is simply an
*additional* mechanism for manipulating Neutron, not a deprecation of any
part of the existing Neutron API. I think Jay's concern and the reason
why he keeps mentioning Nova as the biggest and most important consumer
of Neutron's API stems from an assumption that Nova would need to change
to use the GBP API.

If I've understood the follow on discussions correctly, there's no need for
Nova to use the GBP API at all until/unless the Nova developers see benefit
in it because they can continue to accomplish everything with the existing
API. The GBP API simply provides a more application centric rather than
network centric representation of the same thing.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Jeremy Stanley
On 2014-08-11 08:04:34 -0400 (-0400), Russell Bryant wrote:
 Dang, I'd love to see those numbers.  :-)

Me too. Now that I'm not travelling I'll see if I can find out what
he meant by that.

 Understood.  Some questions ... is building an image that has libvirt
 and qemu pre-installed from source good enough?  It avoids the
 dependency on job runs, but moves it to image build time though, so it
 still exists.

Moving complex stability risks to image creation time still causes
us to potentially fail to update our worker images as often, which
means tests randomly run on increasingly stale systems in some
providers/regions until the issue is noticed, identified and
addressed. That said, we do already compile some things during job
runs today (in particular, library bindings which get install-time
linked by some Python modules).

In reality, depending on more things gathered from different places
on the Internet (be it Git repository sites like GitHub/Bitbucket,
or private package collections) decreases our overall stability far
more than compiling things does.

 If the above still doesn't seem like a workable setup, then I think we
 should just go straight to an image with fedora + virt-preview repo,
 which kind of sounds easier, anyway.

If it's published from EPEL or whatever Fedora's equivalent is, then
that's probably fine. If it's served from a separate site, then that
increases the chances that we run into network issues either at
image build time or job run time. Also, we would want to make sure
whatever solution we settle on is well integrated within DevStack
itself, so that individual developers can recreate these conditions
themselves without a lot of additional work.

One other thing to keep in mind... Fedora's lifecycle is too short
for us to support outside of jobs for our master branches, so this
would not be a solution beyond release time (we couldn't continue to
run these jobs for Juno once released if the solution hinges on
Fedora). Getting the versions we want developers and deployers to
use into Ubuntu 14.04 Cloud Archive and CentOS (RHEL) 7 EPEL on the
other hand would be a much more viable long-term solution.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [keystone] how to generate keystone.conf.sample in havana release?

2014-08-11 Thread Dolph Mathews
Unfortunately, there is no way to generate a sample conf in the Havana
release. The etc/keystone.conf.sample included in Havana was maintained
manually. In icehouse (and master), you can generate a sample configuration
file using tox:

  $ tox -e sample_config

Or run generate_sample.sh directly if you already have all the dependencies
available:

  $ sh tools/config/generate_sample.sh

-- Forwarded message --
From: jazeltq jaze...@163.com
Date: Mon, Aug 11, 2014 at 5:06 AM
Subject: how to generate keystone.conf.sample in havana release?
To: Dolph Mathews dolph.math...@gmail.com


Hello
Is there a way to auto-generate keystone.conf.sample in havana release
keystone?
--
This message was sent from Launchpad by
jazeltq (https://launchpad.net/~jazeltq-k)
using the Contact this team's admins link on the Keystone Drivers team
page
(https://launchpad.net/~keystone-drivers).
For more information see
https://help.launchpad.net/YourAccount/ContactingPeople
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [sahara] Backports for 2014.1.2

2014-08-11 Thread Sergey Lukjanov
There were no proposals for additional backports for sahara 2014.1.2,
so, I'm going to release it later today.

On Thu, Aug 7, 2014 at 10:48 PM, Sergey Lukjanov slukja...@mirantis.com wrote:
 Hey sahara folks,

 I'm going to push 2014.1.2 tag to stable/icehouse branch next week,
 so, please, propose backports before the weekend and ping us to
 backport some sensitive fixes.

 Thanks you!
 --
 Sincerely yours,
 Sergey Lukjanov
 Sahara Technical Lead
 (OpenStack Data Processing)
 Principal Software Engineer
 Mirantis Inc.



-- 
Sincerely yours,
Sergey Lukjanov
Sahara Technical Lead
(OpenStack Data Processing)
Principal Software Engineer
Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Russell Bryant
On 08/11/2014 09:17 AM, Jeremy Stanley wrote:
 On 2014-08-11 08:04:34 -0400 (-0400), Russell Bryant wrote:
 Dang, I'd love to see those numbers.  :-)
 
 Me too. Now that I'm not travelling I'll see if I can find out what
 he meant by that.
 
 Understood.  Some questions ... is building an image that has libvirt
 and qemu pre-installed from source good enough?  It avoids the
 dependency on job runs, but moves it to image build time though, so it
 still exists.
 
 Moving complex stability risks to image creation time still causes
 us to potentially fail to update our worker images as often, which
 means tests randomly run on increasingly stale systems in some
 providers/regions until the issue is noticed, identified and
 addressed. That said, we do already compile some things during job
 runs today (in particular, library bindings which get install-time
 linked by some Python modules).
 
 In reality, depending on more things gathered from different places
 on the Internet (be it Git repository sites like GitHub/Bitbucket,
 or private package collections) decreases our overall stability far
 more than compiling things does.
 
 If the above still doesn't seem like a workable setup, then I think we
 should just go straight to an image with fedora + virt-preview repo,
 which kind of sounds easier, anyway.
 
 If it's published from EPEL or whatever Fedora's equivalent is, then
 that's probably fine. If it's served from a separate site, then that
 increases the chances that we run into network issues either at
 image build time or job run time. Also, we would want to make sure
 whatever solution we settle on is well integrated within DevStack
 itself, so that individual developers can recreate these conditions
 themselves without a lot of additional work.

EPEL is a repo produced by the Fedora project for RHEL and its
derivatives.  The virt-preview repo is hosted on fedorapeople.org, which
is where custom repos live.  I'd say it's more analogous to Ubuntu's PPAs.

https://fedorapeople.org/groups/virt/virt-preview/

 One other thing to keep in mind... Fedora's lifecycle is too short
 for us to support outside of jobs for our master branches, so this
 would not be a solution beyond release time (we couldn't continue to
 run these jobs for Juno once released if the solution hinges on
 Fedora). Getting the versions we want developers and deployers to
 use into Ubuntu 14.04 Cloud Archive and CentOS (RHEL) 7 EPEL on the
 other hand would be a much more viable long-term solution.

Yep, makes sense.

For testing bleeding edge, I've also got my eye on how we could do this
with CentOS.  There is a virt SIG in CentOS that I'm hoping will produce
something similar to Fedora's virt-preview repo, but it's not there yet.
 I'm going to go off and discuss this with the SIG there.

http://wiki.centos.org/SpecialInterestGroup/Virtualization

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [sahara] new client release (0.7.1)

2014-08-11 Thread Sergey Lukjanov
Hey sahara folks,

the latest sahara client release was at Mar 29 (right after the fully
completed renaming process) and so we have a lot of unreleased fixes
and improvements.

You can check diff [0] and launchpad's release page [1].

I'm going to release 0.7.1 this week, so, please propose changes to be
included into the new client release in the next few days.

Thanks.

[0] https://github.com/openstack/python-saharaclient/compare/0.7.0...HEAD
[1] https://launchpad.net/python-saharaclient/+milestone/0.7.1

-- 
Sincerely yours,
Sergey Lukjanov
Sahara Technical Lead
(OpenStack Data Processing)
Principal Software Engineer
Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Jeremy Stanley
On 2014-08-11 08:07:02 -0400 (-0400), Russell Bryant wrote:
 There's a pypi mirror at least.  I'm not sure about mirroring
 other things.

Right, that's a specific solution for mirroring the pypi.python.org
cheeseshop. We've got our (Infra) sights set on mirroring Ubuntu and
CentOS package repositories to similarly reduce the incidence of job
run-time failures we see getting updated packages and indexes from
distro sites and cloud provider mirrors, but those too are
nontrivial efforts and will be a while yet before we have them in
place. Mirroring things is generally complex, since different kinds
of files/data need widely differing retrieval, indexing and caching
solutions--there's no one-size-fits-all option really.

Perhaps another good example is the Fedora qcow2 image we download
and cache on DevStack worker images so that Heat can perform some of
its more complex integration tests... failures encountered when
obtaining that image from dl.fedoraproject.org are (last time I
checked anyway) our most frequent cause of nodepool update problems.
We could set up our own mirror of that file of course, but to some
extent that's still just moving the problem--each additional
mirroring solution is something new we have to monitor, maintain and
troubleshoot so we must ask ourselves whether the increased
management burden from that new complexity is balanced by potential
decreases in management burden found by improving stability in other
parts of the system.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [designate] [neutron] designate and neutron integration

2014-08-11 Thread Miyashita, Kazuhiro
Hi,

I want to ask about neutron and designate integration.
I think dnsmasq fowards DNS request from instance to designate is better.

   ++
   |DNS server(designate)   |
   ++
|
-+--+-- Network1
 |
  ++
  |dnsmasq |
  ++
|
-+--+-- Network2
 |
+-+
|instance |
+-+

Because it's simpler than virtual router connects Network1 and Network2.
If router connects network, instance should know where DNS server is. it is 
complicated.
dnsmasq returns its ip address as dns server in DHCP replay by ordinary, so,
I think good dnsmasq becomes like a gateway to designate.

But, I can't connect dnsmasq to Network1. because of today's neutron design.

Question:
  Does designate design team have a plan such as above integration?
  or other integration design?

*1: Network1 and Network2 are deployed by neutron.
*2: neutron deploys dnsmasq as a dhcp server.
dnsmasq can forward DNS request.

Thanks,

kazuhiro MIYASHITA




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [OSSN 0022] Nova Networking does not enforce security group rules following a soft reboot of an instance

2014-08-11 Thread Nathan Kinder
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Nova Networking does not enforce security group rules following a soft
reboot of an instance
- ---

### Summary ###
In deployments using Nova Networking, security group rules associated
with an instance may not be enforced after a soft reboot. Nova is
designed to apply the configured security group rules to an instance
when certain operations are performed, such as a normal boot operation.
If an operation has been performed that results in the clearing of
security group rules, such as restarting the nova compute service, then
performing a soft reboot of that instance will cause it to be
started without security group rules being applied.

Deployments using Neutron are not impacted.

### Affected Services / Software ###
Nova, Havana, Grizzly

### Discussion ###
In Nova deployments using Nova Networking, security groups are
implemented using iptables, which is used to configure and control
network traffic into Nova instances. When an instance is first booted
using the normal boot method (nova boot instance_id), the security
group rules are applied to that instance.

When an instance is rebooted using the soft reboot method (nova reboot
instance_id), the security group rules are not reapplied since they
should have been already applied when the instance was initially
booted. If the security group rules have not been applied following an
event that resulted in their clearing, such as restarting the compute
service, the instance will be brought up without security group
enforcement. This situation is most likely to arise in cases where the
Nova compute service has been terminated or restarted, which removes
all iptables rules. If a stopped instance is then started by using a
soft reboot, it will not have any security group rules applied. A hard
reboot (nova reboot --hard instance_id) reapplies the security group
rules, so it is not susceptible to this issue.

Depending on the deployment architecture, this could breach security
assumptions and leave an instance vulnerable to network based attacks.

This issue only affects the Havana and Grizzly releases. The Icehouse
release does not allow a stopped instance to be started using a soft
reboot, therefore this issue does not affect the Icehouse release.

### Recommended Actions ###
Do not to use the soft reboot method to start instances from the
stopped state. If instances are in the stopped state, boot using nova
boot instance_id or reboot using nova reboot --hard instance_id
to force the security group rules to be applied.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0022
Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/1316822
OpenStack Security ML : openstack-secur...@lists.openstack.org
OpenStack Security Group : https://launchpad.net/~openstack-ossg
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJT6MtUAAoJEJa+6E7Ri+EV2t4H/RwM5auF4ik9tdZuRNVLI/Rv
Si1SBx+EZA5QOksJ1r476mm2nA9+OnhCZRVnz8bHGiKtzC7eRV82c1OO1q0R/w1/
zHuTd6ZsZaAyBshfv6YoEVoRn6I2duYiR6gRHz/+hfrItt5E+SYBXqFskJazJ6dF
PZBA16qrIPmeir1eDpDvFnpbMEkAMUDKGZNba/DXwEZgdKVdFaWPI3cNIsHVP6bL
fsqMX+7bldB8kBDKfyFleFYqBUI5Anonzrq1tu6TS0vXl4gsfD2LE5LphPTYaiC+
EMypsACSafd19RhpeHS7/I1iVhf/Xa26Bvne0T8HWMLuDd1M5TT9E7nnas45k24=
=1x8I
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Daniel P. Berrange
On Mon, Aug 11, 2014 at 01:45:56PM +, Jeremy Stanley wrote:
 On 2014-08-11 08:07:02 -0400 (-0400), Russell Bryant wrote:
  There's a pypi mirror at least.  I'm not sure about mirroring
  other things.
 
 Right, that's a specific solution for mirroring the pypi.python.org
 cheeseshop. We've got our (Infra) sights set on mirroring Ubuntu and
 CentOS package repositories to similarly reduce the incidence of job
 run-time failures we see getting updated packages and indexes from
 distro sites and cloud provider mirrors, but those too are
 nontrivial efforts and will be a while yet before we have them in
 place. Mirroring things is generally complex, since different kinds
 of files/data need widely differing retrieval, indexing and caching
 solutions--there's no one-size-fits-all option really.

If there are specific things we could do to libvirt.org / qemu.org
download sites, to make mirroring easier or more reliable for OpenStack,
we could certainly explore options there, since we know the right people
involved in both projects. 

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Samuel Bercovici
Hi,

Validations such as timeout  delay should be performed on the API level 
before it reaches the driver.

For a configuration tree (lb, listeners, pools, etc.), there should be one 
provider.
Having provider defined in multiple places does not make sense.


-San.


From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]
Sent: Monday, August 11, 2014 2:43 PM
To: OpenStack Development Mailing List (openstack-dev@lists.openstack.org)
Subject: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver 
interface on every API request

Hi:

Continuing from last week's LBaaS meeting...

Currently an entity cannot be sent to driver unless it is linked to 
loadbalancer because loadbalancer is the root object and driver information is 
only available with loadbalancer.

The request to the driver is delayed because of which error propagation becomes 
tricky.

Let's say a monitor was configured with timeout  delay there would be no error 
then.
When a listener is configured there will be a monitor creation/deployment error 
like timeout configured greater than delay.

Unless the error is very clearly crafted the user won't be able to understand 
the error.

I am half-heartedly OK with current approach.

But, I would prefer Brandon's Solution - make provider an attribute in each of 
the entities to get rid of this problem.

What do others think?

Thanks,
Vijay V.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

2014-08-11 Thread Jeremy Stanley
On 2014-08-11 14:56:27 +0100 (+0100), Daniel P. Berrange wrote:
 If there are specific things we could do to libvirt.org / qemu.org
 download sites, to make mirroring easier or more reliable for
 OpenStack, we could certainly explore options there, since we know
 the right people involved in both projects.

Mirroring is generally a reaction to stability issues more than
anything. Providing those files from multiple locations behind a
global cache/load-balancing solution (maybe a CDN) to ensure
stability would likely help prevent us from needing to have yet
another one-off mirror for a handful of files. It's worth adding to
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
and discussing tomorrow, if you're around, so we can be sure to
get input from more of the Infra team.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone] how to generate conf sample in havana?

2014-08-11 Thread Dolph Mathews
See here:
http://lists.openstack.org/pipermail/openstack-dev/2014-August/042573.html


On Mon, Aug 11, 2014 at 3:53 AM, Li Tianqing jaze...@163.com wrote:

 Hello,
Right now, i customerized keystone to meet our requeriments. I want to
 add some configuration options in keystone.conf.sample. But, i
 do not find generate_sample.sh in directory tools in Havana keystone. So i
 want to know how to generate conf sample in havana keystone.
 I know, i can write directly to keystone.conf.sample, but should there be
 something like generate_sample.sh in keystone?



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] PCI support

2014-08-11 Thread Robert Li (baoli)
Gary,

Cisco is adding it in our CI testbed. I guess that mlnx is doing the same for 
their MD as well.

—Robert

On 8/11/14, 9:05 AM, Gary Kotton 
gkot...@vmware.commailto:gkot...@vmware.com wrote:

Hi,
At the moment all of the drivers are required CI support. Are there any plans 
regarding the PIC support. I understand that this is something that requires 
specific hardware. Are there any plans to add this?
Thanks
Gary
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Is network ordering of vNICs guaranteed?

2014-08-11 Thread CARVER, PAUL
Armando M. [mailto:arma...@gmail.com] wrote:

On 9 August 2014 10:16, Jay Pipes jaypi...@gmail.com wrote:
Paul, does this friend of a friend have a reproduceable test
script for this?

We would also need to know the OpenStack release where this issue manifest
itself. A number of bugs have been raised in the past around this type of
issue, and the last fix I recall is this one:

https://bugs.launchpad.net/nova/+bug/1300325

It's possible that this might have regressed, though.

The reason I called it friend of a friend is because I think the info
has filtered through a series of people and is not firsthand observation.
I'll ask them to track back to who actually observed the behavior, how
long ago, and with what version.

It could be a regression, or it could just be old info that people have
continued to assume is true without realizing it was considered a bug
all along and has been fixed.

Thanks! The moment I first heard it my first reaction was that it was
almost certainly a bug and had probably already been fixed.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [keystone] how to generate keystone.conf.sample in havana release?

2014-08-11 Thread Li Tianqing
Ok, thank you all the same.

At 2014-08-11 09:31:19, Dolph Mathews dolph.math...@gmail.com wrote:

Unfortunately, there is no way to generate a sample conf in the Havana release. 
The etc/keystone.conf.sample included in Havana was maintained manually. In 
icehouse (and master), you can generate a sample configuration file using tox:


  $ tox -e sample_config


Or run generate_sample.sh directly if you already have all the dependencies 
available:


  $ sh tools/config/generate_sample.sh


-- Forwarded message --
From: jazeltqjaze...@163.com
Date: Mon, Aug 11, 2014 at 5:06 AM
Subject: how to generate keystone.conf.sample in havana release?
To: Dolph Mathews dolph.math...@gmail.com


Hello
Is there a way to auto-generate keystone.conf.sample in havana release
keystone?
--
This message was sent from Launchpad by
jazeltq (https://launchpad.net/~jazeltq-k)
using the Contact this team's admins link on the Keystone Drivers team page
(https://launchpad.net/~keystone-drivers).
For more information see
https://help.launchpad.net/YourAccount/ContactingPeople


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Nova] PCI support

2014-08-11 Thread Gary Kotton
Thanks for the update.

From: Robert Li (baoli) ba...@cisco.commailto:ba...@cisco.com
Reply-To: OpenStack List 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Monday, August 11, 2014 at 5:08 PM
To: OpenStack List 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [Nova] PCI support

Gary,

Cisco is adding it in our CI testbed. I guess that mlnx is doing the same for 
their MD as well.

-Robert

On 8/11/14, 9:05 AM, Gary Kotton 
gkot...@vmware.commailto:gkot...@vmware.com wrote:

Hi,
At the moment all of the drivers are required CI support. Are there any plans 
regarding the PIC support. I understand that this is something that requires 
specific hardware. Are there any plans to add this?
Thanks
Gary
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

2014-08-11 Thread Jay Pipes

On 08/10/2014 10:36 PM, Jay Lau wrote:

I was asking this because I got a -2 for
https://review.openstack.org/109505 , just want to know why this new
term metadetails was invented when we already have details,
metadata, system_metadata, instance_metadata, and properties (on
images and volumes).


As the person who -2'd the review, I'm thankful you raised this issue on 
the ML, Jay. Much appreciated.


Eagerly awaiting answers,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

2014-08-11 Thread Clint Byrum
Excerpts from Steve Baker's message of 2014-08-10 15:33:26 -0700:
 On 02/08/14 04:07, Allison Randal wrote:
  A few of us have been independently experimenting with Ansible as a
  backend for TripleO, and have just decided to try experimenting
  together. I've chatted with Robert, and he says that TripleO was always
  intended to have pluggable backends (CM layer), and just never had
  anyone interested in working on them. (I see it now, even in the early
  docs and talks, I guess I just couldn't see the forest for the trees.)
  So, the work is in line with the overall goals of the TripleO project.
 
  We're starting with a tiny scope, focused only on updating a running
  TripleO deployment, so our first work is in:
 
  - Create an Ansible Dynamic Inventory plugin to extract metadata from Heat
  - Improve/extend the Ansible nova_compute Cloud Module (or create a new
  one), for Nova rebuild
  - Develop a minimal handoff from Heat to Ansible, particularly focused
  on the interactions between os-collect-config and Ansible
 
  We're merging our work in this repo, until we figure out where it should
  live:
 
  https://github.com/allisonrandal/tripleo-ansible
 
  We've set ourselves one week as the first sanity-check to see whether
  this idea is going anywhere, and we may scrap it all at that point. But,
  it seems best to be totally transparent about the idea from the start,
  so no-one is surprised later.
 
 Having pluggable backends for configuration seems like a good idea, and
 Ansible is a great choice for the first alternative backend.
 

TripleO is intended to be loosely coupled for many components, not just
in-instance configuration.

 However what this repo seems to be doing at the moment is bypassing heat
 to do a stack update, and I can only assume there is an eventual goal to
 not use heat at all for stack orchestration too.


 Granted, until blueprint update-failure-recovery lands[1] then doing a
 stack-update is about as much fun as russian roulette. But this effort
 is tactical rather than strategic, especially given TripleO's mission
 statement.
 

We intend to stay modular. Ansible won't replace Heat from end to end.

Right now we're stuck with an update that just doesn't work. It isn't
just about update-failure-recovery, which is coming along nicely, but
it is also about the lack of signals to control rebuild, poor support
for addressing machines as groups, and unacceptable performance in
large stacks.

We remain committed to driving these things into Heat, which will allow
us to address these things the way a large scale operation will need to.

But until we can land those things in Heat, we need something more
flexible like Ansible to go around Heat and do things in the exact
order we need them done. Ansible doesn't have a REST API, which is a
non-starter for modern automation, but the need to control workflow is
greater than the need to have a REST API at this point.

 If I were to use Ansible for TripleO configuration I would start with
 something like the following:
 * Install an ansible software-config hook onto the image to be triggered
 by os-refresh-config[2][3]
 * Incrementally replace StructuredConfig resources in
 tripleo-heat-templates with SoftwareConfig resources that include the
 ansible playbooks via get_file
 * The above can start in a fork of tripleo-heat-templates, but can
 eventually be structured using resource providers so that the deployer
 chooses what configuration backend to use by selecting the environment
 file that contains the appropriate config resources
 
 Now you have a cloud orchestrated by heat and configured by Ansible. If
 it is still deemed necessary to do an out-of-band update to the stack
 then you're in a much better position to do an ansible push, since you
 can use the same playbook files that heat used to bring up the stack.
 

That would be a good plan if we wanted to fix issues with os-*-config,
but that is the opposite of reality. We are working around Heat
orchestration issues with Ansible.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Is network ordering of vNICs guaranteed?

2014-08-11 Thread Jay Pipes

Thanks, Paul!

On 08/11/2014 10:10 AM, CARVER, PAUL wrote:

Armando M. [mailto:arma...@gmail.com] wrote:


On 9 August 2014 10:16, Jay Pipes jaypi...@gmail.com wrote:



Paul, does this friend of a friend have a reproduceable test



script for this?



We would also need to know the OpenStack release where this issue manifest



itself. A number of bugs have been raised in the past around this type of



issue, and the last fix I recall is this one:







https://bugs.launchpad.net/nova/+bug/1300325







It's possible that this might have regressed, though.


The reason I called it friend of a friend is because I think the info

has filtered through a series of people and is not firsthand observation.

I'll ask them to track back to who actually observed the behavior, how

long ago, and with what version.

It could be a regression, or it could just be old info that people have

continued to assume is true without realizing it was considered a bug

all along and has been fixed.

Thanks! The moment I first heard it my first reaction was that it was

almost certainly a bug and had probably already been fixed.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

2014-08-11 Thread Dan Smith
 As the person who -2'd the review, I'm thankful you raised this issue on
 the ML, Jay. Much appreciated.

The metadetails term isn't being invented in this patch, of course. I
originally complained about the difference when this was being added:

https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contrib/server_groups.py,cm

As best I can tell, the response in that patch set about why it's being
translated is wrong (backwards). I expect that the API extension at the
time called it metadetails and they decided to make the object the
same and do the translation there.

From what I can tell, the actual server_group API extension that made it
into the tree never got the ability to set/change/etc the
metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
wrongly.

If we care to have this functionality, then I propose we change the
attribute on the object (we can handle this with versioning) and reflect
it as metadata in the API.

However, I have to ask: do we really need another distinct metadata
store attached to server_groups? If not, how about we just remove it
from the database and the object, clean up the bit of residue that is
still in the API extension and be done with it?

--Dan



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

2014-08-11 Thread Jay Pipes

On 08/11/2014 11:06 AM, Dan Smith wrote:

As the person who -2'd the review, I'm thankful you raised this issue on
the ML, Jay. Much appreciated.


The metadetails term isn't being invented in this patch, of course. I
originally complained about the difference when this was being added:

https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contrib/server_groups.py,cm

As best I can tell, the response in that patch set about why it's being
translated is wrong (backwards). I expect that the API extension at the
time called it metadetails and they decided to make the object the
same and do the translation there.

 From what I can tell, the actual server_group API extension that made it
into the tree never got the ability to set/change/etc the
metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
wrongly.

If we care to have this functionality, then I propose we change the
attribute on the object (we can handle this with versioning) and reflect
it as metadata in the API.

However, I have to ask: do we really need another distinct metadata
store attached to server_groups?


No.

 If not, how about we just remove it

from the database and the object, clean up the bit of residue that is
still in the API extension and be done with it?


+1

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

2014-08-11 Thread Zane Bitter

On 11/08/14 10:46, Clint Byrum wrote:

Right now we're stuck with an update that just doesn't work. It isn't
just about update-failure-recovery, which is coming along nicely, but
it is also about the lack of signals to control rebuild, poor support
for addressing machines as groups, and unacceptable performance in
large stacks.


Are there blueprints/bugs filed for all of these issues?

-ZB

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree

2014-08-11 Thread Robert Kukura


On 8/11/14, 4:52 AM, Thierry Carrez wrote:

gustavo panizzo (gfa) wrote:

only one think i didn't like it

why all url,api, etc has to include the word 'preview'?
i imagine that i would be consuming the new feature using heat, puppet,
local scripts, custom horizon, whatever. Why do you make me to change
all them when the feature moves out of preview? it could be a lot of
rework (for consumers) without gain. I would totally support other big
fat warnings everywhere (logs, documentation, startup log of
neutron-server) but don't change the API if is not necessary

I see two issues with this proposal: the first one is what Gustavo just
said: the use of the preview package/configoption/API creates friction
when the feature needs to go mainstream (package renaming, change in
configuration for deployers, change in API calls for users...).

Hi Thierry,

I completely agree with you and with Gustavo that mangling the REST 
URIs to include preview may have more cost (i.e. friction when the API 
becomes stable) than benefit. I'm happy to drop that particular part of 
the proposal. The email was intended to kick off discussion of these 
sorts of details.


My understanding is that the goal is to make it easy for people to try
the preview feature, and keeping the experimental feature in-tree is
seen as simpler to experiment with. But the pain from this friction imho
outweighs the pain of deploying an out-of-tree plugin for deployers.
I agree out-of-tree is a better option for truly experimental features. 
This in-tree stabilization is intended for a beta release, as opposed to 
a prototype.


The second issue is that once the feature is in preview in tree, it
moves the responsibility/burden of making it official (or removed) to
the core developers (as Salvatore mentioned). I kind of like the
approach where experimental features are developed in faster iterations
out-of-tree and when they are celebrated by experimenters and are ready
to be stable-supported, they are moved in tree.
I don't think we are really disagreeing here. There are clearly 
situations where rapid iteration out-of-tree, without the burden of the 
core review process, is most appropriate. But this proposal is intended 
for features that are on the cusp of being declared stable, rather than 
for experimentation. The intent is absolutely to have all changes to the 
code go through the regular core review process during this 
stabilization phase. This enables the feature to be fully reviewed and 
integrated (also including CLIs, Horizon and Heat support, 
documentation, etc.) at the point when the decision is made that no 
further incompatible API changes will be needed. Once the feature is 
declared stable, from the end-user perspective, its just a matter of 
removing the preview label. Moving the feature's code from the preview 
subtree to its normal locations in the tree will not effect most users 
or operators.


Note that the GBP team had implemented a proof-of-concept prior to the 
start of the Juno cycle out-of-tree. Our initial plan was to get this 
PoC code reviewed and merged at the beginning of Juno, and then 
iteratively improve it throughout the cycle. But we got a lot of 
resistance to the idea of merging a large body of code that had been 
developed outside the Neutron development and review process. We've 
instead had to break it into multiple pieces, and make sure each of 
those is production ready, to have any chance of getting through the 
review process during Juno.  Its not really clear that something 
significant developed externally can ever be moved in tree, at least 
without major upheaval, including incompatible API changes, as it goes 
through the review/merge process.


Finally, consider that many interesting potential features for Neutron 
involve integrations with external back-ends, such as ODL or 
vendor-specific devices or controllers, along with a reference 
implementation that doesn't depend on external systems. To really 
validate that the API, the model, and the driver framework code for a 
new feature are all stable, it is necessary to implement and deploy 
several of these back-end integrations along with the reference 
implementation. But vendors may not be willing to invest substantially 
in this integration effort without the assurances about the quality and 
relative stability of the interfaces involved that comes from the core 
review process, and without the clear path to market that comes with 
in-tree development of an approved blueprint targeted at a specific 
Neutron release.


-Bob


Regards,




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Manila] Incubation request

2014-08-11 Thread Swartzlander, Ben
I just saw the agenda for tomorrow's TC meeting and we're on it. I plan to be 
there.

https://wiki.openstack.org/wiki/Meetings#Technical_Committee_meeting

-Ben


From: Swartzlander, Ben [mailto:ben.swartzlan...@netapp.com]
Sent: Monday, July 28, 2014 9:53 PM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Manila] Incubation request

Manila has come a long way since we proposed it for incubation last autumn. 
Below are the formal requests.

https://wiki.openstack.org/wiki/Manila/Incubation_Application
https://wiki.openstack.org/wiki/Manila/Program_Application

Anyone have anything to add before I forward these to the TC?

-Ben Swartzlander

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Fuel] 6.0 blueprints cleanup

2014-08-11 Thread Dmitry Pyzhov
We've moved all blueprints from 6.0 to 'next' milestone. It has been done
in order to better view of stuff that we really want to implement in 6.0.

Feature freeze for 6.0 release is planned to 18th of September. If you are
going to merge your blueprint before that date, you can move it to 6.0
milestone and 6.0.x series. But blueprint must have fixed scope and must be
assigned to person who will lead this activity.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] 6.0 blueprints cleanup

2014-08-11 Thread Dmitry Borodaenko
All,

Please refer to this email for a list of information that has to be
populated in a blueprint before it can be assigned and scheduled:
http://lists.openstack.org/pipermail/openstack-dev/2014-August/042042.html

Please don't schedule a blueprint to 6.0 until it has these details
and its assignees are confirmed.

On Mon, Aug 11, 2014 at 8:30 AM, Dmitry Pyzhov dpyz...@mirantis.com wrote:
 We've moved all blueprints from 6.0 to 'next' milestone. It has been done in
 order to better view of stuff that we really want to implement in 6.0.

 Feature freeze for 6.0 release is planned to 18th of September. If you are
 going to merge your blueprint before that date, you can move it to 6.0
 milestone and 6.0.x series. But blueprint must have fixed scope and must be
 assigned to person who will lead this activity.



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Dmitry Borodaenko

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Brandon Logan
Yeah what I meant was the only solution I could come up with so that the
driver gets passed every call is to have the every entity have a
provider.  I do believe this is a bit cumbersome for a user, and extra
validation would be needed to verify that two entities linked together
cannot have different providers, but that would be pretty easy.  In my
opinion, it'd be a bit weird to have them all have a provider.  However,
there are some pros to it such as:

1) Driver always gets the create, update, and delete calls.
2) If drivers support a varying range of values for certain entity
attributes, that validation can be caught immediately if thats something
people wanted.
3) Will remove the necessity of a DEFERRED status for some drivers
(This also brings up a CON, in that some drivers may use DEFERRED and
some may not, which leads to an inconsistent UX).
4) Status management in some drivers will become a bit easier.

Still I don't think it is something that should be done because having
the user give a provider for every entity is a bit cumbersome.  Though
if enough people want this then a larger discussion about it should
probably happen.

Thanks,
Brandon

On Mon, 2014-08-11 at 13:57 +, Samuel Bercovici wrote:
 Hi,
 
  
 
 Validations such as “timeout  delay” should be performed on the API
 level before it reaches the driver.
 
  
 
 For a configuration tree (lb, listeners, pools, etc.), there should be
 one provider. 
 
 Having provider defined in multiple places does not make sense.
 
  
 
  
 
 -San.
 
  
 
  
 
 From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com] 
 Sent: Monday, August 11, 2014 2:43 PM
 To: OpenStack Development Mailing List
 (openstack-dev@lists.openstack.org)
 Subject: [openstack-dev] [Neutron][LBaaS] Continuing on Calling
 driver interface on every API request
 
 
  
 
 Hi:
 
  
 
 Continuing from last week’s LBaaS meeting…
 
  
 
 Currently an entity cannot be sent to driver unless it is linked to
 loadbalancer because loadbalancer is the root object and driver
 information is only available with loadbalancer. 
 
  
 
 The request to the driver is delayed because of which error
 propagation becomes tricky.
 
  
 
 Let’s say a monitor was configured with timeout  delay there would be
 no error then.
 
 When a listener is configured there will be a monitor
 creation/deployment error like “timeout configured greater than
 delay”.
 
  
 
 Unless the error is very clearly crafted the user won’t be able to
 understand the error.
 
  
 
 I am half-heartedly OK with current approach. 
 
  
 
 But, I would prefer Brandon’s Solution – make provider an attribute in
 each of the entities to get rid of this problem. 
 
  
 
 What do others think?
 
  
 
 Thanks,
 
 Vijay V.
 
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] devstack local.conf file

2014-08-11 Thread Nikesh Kumar Mahalka
Hi,
I have gone through devstack links.
They are not clear like openstack.org documents.


For Example:
when i am using below local.conf file in devstack,hp_msa_driver is not
coming in enabled_backends in cinder.conf after running stack.sh.

[[local|localrc]]
ADMIN_PASSWORD=vedams123
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=ADMIN
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.2.170/29
HOST_IP=192.168.2.151
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
SYSLOG=True
SYSLOG_HOST=$HOST_IP
SYSLOG_PORT=516
TEMPEST_VOLUME_DRIVER=hp_msa_fc
TEMPEST_VOLUME_VENDOR=Hewlett-Packard
TEMPEST_STORAGE_PROTOCOL=FC


[[post-config|$CINDER_CONF]]
[hp_msa_driver]
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver
san_ip = 192.168.2.192
san_login = manage
san_password =!manage
volume_backend_name=HPMSA_FC


[lvmdriver-1]
volume_group=stack-volumes-1
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=LVM_iSCSI



*I am getting below cinder.conf file after running stack.sh script*

[keystone_authtoken]
auth_uri = http://192.168.2.151:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = vedams123
admin_user = cinder
admin_tenant_name = service
cafile =
identity_uri = http://192.168.2.151:35357

[DEFAULT]
rabbit_password = vedams123
rabbit_hosts = 192.168.2.151
rpc_backend = cinder.openstack.common.rpc.impl_kombu
use_syslog = True
*default_volume_type = lvmdriver-1*
*enabled_backends = lvmdriver-1*
enable_v1_api = true
periodic_interval = 60
lock_path = /opt/stack/data/cinder
state_path = /opt/stack/data/cinder
osapi_volume_extension = cinder.api.contrib.standard_extensions
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_config = /etc/cinder/api-paste.ini
sql_connection = mysql://root:vedams123@127.0.0.1/cinder?charset=utf8
iscsi_helper = tgtadm
my_ip = 192.168.2.151
verbose = True
debug = True
auth_strategy = keystone

[lvmdriver-1]
volume_group = stack-volumes-1
volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name = LVM_iSCSI

[hp_msa_driver]
volume_backend_name = HPMSA_FC
san_password = !manage
san_login = manage
san_ip = 192.168.2.192
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver



*Then i analyzed source code of stack.sh,and added in local.conf this line:*
*CINDER_ENABLED_BACKENDS=hp_msa:hp_msa_driver,lvm:lvmdriver-1*


Now i am getting hp_msa_fc in cinder.conf in enabled_backends



Regards
Nikesh
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Doug Wiegley
Hi all,

 Validations such as ³timeout  delay² should be performed on the API
level before it reaches the driver.
For a configuration tree (lb, listeners, pools, etc.), there should be one
provider.

You¹re right, but I think the point of Vijay¹s example was to highlight
the combo error problem with populating all of the driver objects at once
(in short, the driver interface isn¹t well suited to that model.)  That
his one example can be covered by API validators is irrelevant.  Consider
a backend that does not support APP_COOKIE¹s, or HTTPS_TERMINATED (but has
multiple listeners) instead.  Should the entire load balancer create fail,
or should it offer degraded service?  Do all drivers have to implement a
transaction rollback; wait, the interface makes that very hard.  That¹s
his point.  The driver is no longer just glue code between interfaces;
it¹s now a mini-object error handler.


 Having provider defined in multiple places does not make sense.

Channeling Brandon, who can yell if I get this wrong, the point is not to
have a potentially different provider on each object.  It¹s to allow a
provider to be assigned when the first object in the tree is created, so
that future related objects will always get routed to the same provider.
Not knowing which provider should get all the objects is why we have to
wait until we see a LoadBalancer object.


All of this sort of edge case nonsense is because we (the royal we, the
community), wanted all load balancer objects to be ³root² objects, even
though only one of them is an actual root today, to support many-to-many
relationships among all of them, at some future date, without an interface
change.  If my bias is showing that I¹m not a fan of adding this
complexity for that, I¹m not surprised.

Thanks,
doug


On 8/11/14, 7:57 AM, Samuel Bercovici samu...@radware.com wrote:

Hi,
 
Validations such as ³timeout  delay² should be performed on the API
level before it reaches the driver.
 
For a configuration tree (lb, listeners, pools, etc.), there should be
one provider.

Having provider defined in multiple places does not make sense.
 
 
-San.
 
 
From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]

Sent: Monday, August 11, 2014 2:43 PM
To: OpenStack Development Mailing List (openstack-dev@lists.openstack.org)
Subject: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver
interface on every API request


 
Hi:
 
Continuing from last week¹s LBaaS meetingŠ
 
Currently an entity cannot be sent to driver unless it is linked to
loadbalancer because loadbalancer is the root object and driver
information is only available with loadbalancer.

 
The request to the driver is delayed because of which error propagation
becomes tricky.
 
Let¹s say a monitor was configured with timeout  delay there would be no
error then.
When a listener is configured there will be a monitor creation/deployment
error like ³timeout configured greater than delay².
 
Unless the error is very clearly crafted the user won¹t be able to
understand the error.
 
I am half-heartedly OK with current approach.

 
But, I would prefer Brandon¹s Solution ­ make provider an attribute in
each of the entities to get rid of this problem.

 
What do others think?
 
Thanks,
Vijay V.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Which program for Rally

2014-08-11 Thread David Kranz

On 08/06/2014 05:48 PM, John Griffith wrote:
I have to agree with Duncan here.  I also don't know if I fully 
understand the limit in options.  Stress test seems like it 
could/should be different (again overlap isn't a horrible thing) and I 
don't see it as siphoning off resources so not sure of the issue. 
 We've become quite wrapped up in projects, programs and the like 
lately and it seems to hinder forward progress more than anything else.


I'm also not convinced that Tempest is where all things belong, in 
fact I've been thinking more and more that a good bit of what Tempest 
does today should fall more on the responsibility of the projects 
themselves.  For example functional testing of features etc, ideally 
I'd love to have more of that fall on the projects and their 
respective teams.  That might even be something as simple to start as 
saying if you contribute a new feature, you have to also provide a 
link to a contribution to the Tempest test-suite that checks it. 
 Sort of like we do for unit tests, cross-project tracking is 
difficult of course, but it's a start.  The other idea is maybe 
functional test harnesses live in their respective projects.


Honestly I think who better to write tests for a project than the 
folks building and contributing to the project.  At some point IMO the 
QA team isn't going to scale.  I wonder if maybe we should be thinking 
about proposals for delineating responsibility and goals in terms of 
functional testing?




All good points. Your last paragraph was discussed by the QA team 
leading up to and at the Atlanta summit. The conclusion was that the 
api/functional tests focused on a single project should be part of that 
project. As Sean said, we can envision there being half (or some other 
much smaller number) as many such tests in tempest going forward.


Details are under discussion, but the way this is likely to play out is 
that individual projects will start by creating their own functional 
tests outside of tempest. Swift already does this and neutron seems to 
be moving in that direction. There is a spec to break out parts of 
tempest 
(https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst) 
into a library that might be used by projects implementing functional 
tests.


Once a project has sufficient functional testing, we can consider 
removing its api tests from tempest. This is a bit tricky because 
tempest needs to cover *all* cross-project interactions. In this 
respect, there is no clear line in tempest between scenario tests which 
have this goal explicitly, and api tests which may also involve 
interactions that might not be covered in a scenario. So we will need a 
principled way to make sure there is complete cross-project coverage in 
tempest with a smaller number of api tests.


 -David
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] 6.0 blueprints cleanup

2014-08-11 Thread Mike Scherbakov
+2, yes please


On Mon, Aug 11, 2014 at 7:42 PM, Dmitry Borodaenko dborodae...@mirantis.com
 wrote:

 All,

 Please refer to this email for a list of information that has to be
 populated in a blueprint before it can be assigned and scheduled:
 http://lists.openstack.org/pipermail/openstack-dev/2014-August/042042.html

 Please don't schedule a blueprint to 6.0 until it has these details
 and its assignees are confirmed.

 On Mon, Aug 11, 2014 at 8:30 AM, Dmitry Pyzhov dpyz...@mirantis.com
 wrote:
  We've moved all blueprints from 6.0 to 'next' milestone. It has been
 done in
  order to better view of stuff that we really want to implement in 6.0.
 
  Feature freeze for 6.0 release is planned to 18th of September. If you
 are
  going to merge your blueprint before that date, you can move it to 6.0
  milestone and 6.0.x series. But blueprint must have fixed scope and must
 be
  assigned to person who will lead this activity.
 
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 



 --
 Dmitry Borodaenko

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Mike Scherbakov
#mihgen
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

2014-08-11 Thread Gary Kotton


On 8/11/14, 6:06 PM, Dan Smith d...@danplanet.com wrote:

 As the person who -2'd the review, I'm thankful you raised this issue on
 the ML, Jay. Much appreciated.

The metadetails term isn't being invented in this patch, of course. I
originally complained about the difference when this was being added:

https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contr
ib/server_groups.py,cm

As best I can tell, the response in that patch set about why it's being
translated is wrong (backwards). I expect that the API extension at the
time called it metadetails and they decided to make the object the
same and do the translation there.

From what I can tell, the actual server_group API extension that made it
into the tree never got the ability to set/change/etc the
metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
wrongly.

If we care to have this functionality, then I propose we change the
attribute on the object (we can handle this with versioning) and reflect
it as metadata in the API.

However, I have to ask: do we really need another distinct metadata
store attached to server_groups? If not, how about we just remove it
from the database and the object, clean up the bit of residue that is
still in the API extension and be done with it?

The initial version of the feature did not make use of this. The reason
was that we chose for a very
Limited subset to be used, that is, affinity and anti affinity. Moving
forwards we would like to implement
A number of different policies with this. We can drop it at the moment due
to the fact that it is not used.

I think that Yathi may be using this for the constrain scheduler. But I am
not 100% sure.


--Dan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][LBaaS] Use cases with regards to VIP and routers

2014-08-11 Thread Doug Wiegley
Hi Susanne,

While there are a few operators involved with LBaaS that would have good
input, you might want to also ask this on the non-dev mailing list, for a
larger sample size.

Thanks,
doug

On 8/11/14, 3:05 AM, Susanne Balle sleipnir...@gmail.com wrote:

Gang,
I was asked the following questions around our Neutron LBaaS use cases:
1.  Will there be a scenario where the ³VIP² port will be in a different
Node, from all the Member ³VMs² in a pool.


2.  Also how likely is it for the LBaaS configured subnet to not have a
³router² and just use the ³extra_routes²
 option.
3.  Is there a valid use case where customers will be using the
³extra_routes² with subnets instead of the ³routers².
 ( It would be great if you have some use case picture for this).
Feel free to chime in here and I'll summaries the answers.
Regards Susanne



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron] [third-party] Freescale CI log site is being blocked

2014-08-11 Thread trinath.soman...@freescale.com
Hi-

Today I contacted the service provider regarding the malware in the website. 
Got a response that website is truly functional and malware free.

I too have verified all the directories, subdirectories and log files manually 
for any malware injected into the website. I have detected none.

There is no anonymous login enabled for FTP or cPanel login to the server. The 
FTP is protected with Strong Passcode.

This is an update from my end. 

Like all other CI's now normal browsing of logs is available for view.

The CI is made down to take the above explained changes to take place. Now the 
CI is active running the jobs.

All the old logs are present as in place. 

You may browse the old logs using the url 

For ML2 Mechanism driver : 
http://fslopenstackci.com/{change_number}/{change_patchset}/Freescale-ML2-Mechanism-Driver
For FWaaS Plugin : 
http://fslopenstackci.com/{change_number}/{change_patchset}/Freescale-FWaaS-Plugin

Now I have updated the CI to create a BUILD directory as well to showcase logs 
for rechecks.

With this new change the log URL will be 

For ML2 Mechanism driver : 
http://fslopenstackci.com/{build_number}/{change_number}/{change_patchset}/Freescale-ML2-Mechanism-Driver
For FWaaS Plugin : 
http://fslopenstackci.com/{build_number}/{change_number}/{change_patchset}/Freescale-FWaaS-Plugin


Hi Mestery-

Kindly please verify the access to the site. 

Also, if it's still in blocking mode, kindly mail me the logs with Cisco WSA to 
verify the reason behind this blocking.

Kindly help me with your review on my code at 
https://review.openstack.org/#/c/109659/

Thanking  you all.

--
Trinath Somanchi - B39208
trinath.soman...@freescale.com | Mob: +91 9866 235 130

-Original Message-
From: Anita Kuno [mailto:ante...@anteaya.info] 
Sent: Saturday, August 09, 2014 11:09 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] [third-party] Freescale CI log site is 
being blocked

On 08/08/2014 11:27 PM, trinath.soman...@freescale.com wrote:
 Thanks anita for the reply.
 
 Previously the existing server is accessible by kyle. But now its not being 
 accessible. 
 
 For the paid hosting I have its administered by godaddy
If you are paying godaddy to administer the server, have you asked them why one 
of your users has acknowledged your site is blacklisted by Cisco WSA appliances?

If you are paying them to administer your server, answering your question falls 
within their job.

You need to find out the reason behind Cisco security blocking, that is what I 
am asking you to do. It if fine if you don't know, but it is your 
responsibility to find out.

Thanks Trinath,
Anita.

 and the FTP is only accessed by Jenkins. 
 
 I can try relocating FTP web based file browser script and provide a normal 
 view of files. 
 
 Don't know the reason behind Cisco Security blocking the access where it has 
 given access to view the website before.
 
 Thanks a lot again for the brief email.
 
 
 --
 Trinath Somanchi - B39208
 trinath.soman...@freescale.com | extn: 4048
 
 -Original Message-
 From: Anita Kuno [mailto:ante...@anteaya.info]
 Sent: Saturday, August 09, 2014 10:21 AM
 To: openstack-dev@lists.openstack.org
 Subject: Re: [openstack-dev] [neutron] [third-party] Freescale CI log 
 site is being blocked
 
 On 08/08/2014 10:06 PM, trinath.soman...@freescale.com wrote:
 Hi Sumit-

 When I try to paste a large log text into paste.openstack, It is giving me 
 image verification and says its spam.
 Let's not confuse paste.openstack.org's spam blocker from spam blockers on 
 servers. They are two separate functionalities and the conversation does not 
 move forward if we try to pretend they are the same thing or even remotely 
 related, which they are not.
 
 If you recall, Trinath, the first server you had got hacked since you had not 
 hardened it appropriately. Having hosting via go daddy or any other paid 
 hosting service does not absolve you of the responsibility of having a well 
 maintained server. If you need help maintaining your server, I suggest you 
 contract a server administrator to advise you or do the work. We have to 
 assume a certain level of competence here, due to the responsibility involved 
 I don't think you are going to get many responses to questions if you don't 
 know how to maintain your server.
 This isn't really the the place to ask. Running your third party ci system 
 and copying the logs, sure this is the place, basic server maintenance is 
 your responsibility.
 
 If you recall, a basic evaluation of your server logs told you you had been 
 hacked the last time. This might be a place to start now.
 
 In any case, please maintain your server and please address Kyle's concerns.
 
 Thank you Trianth,
 Anita.

 I don't know why its taken as spam/malware. It's a paid hosting I had from 
 GODADDY.

 --
 Trinath Somanchi - B39208
 trinath.soman...@freescale.com | extn: 4048

 -Original Message-
 From: Sumit Naiksatam 

Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Samuel Bercovici
Hi Doug,

In some implementations Driver !== Device. I think this is also true for HA 
Proxy.
This might mean that there is a difference between creating a logical object 
and when there is enough information to actually schedule/place this into a 
device.
The ability to express such errors (detecting an error on a logical object 
after it was created but when it actually get scheduled) should be discussed 
and addressed anyway.

-Sam.


-Original Message-
From: Doug Wiegley [mailto:do...@a10networks.com] 
Sent: Monday, August 11, 2014 6:55 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver 
interface on every API request

Hi all,

 Validations such as ³timeout  delay² should be performed on the API 
level before it reaches the driver.
For a configuration tree (lb, listeners, pools, etc.), there should be one 
provider.

You¹re right, but I think the point of Vijay¹s example was to highlight the 
combo error problem with populating all of the driver objects at once (in 
short, the driver interface isn¹t well suited to that model.)  That his one 
example can be covered by API validators is irrelevant.  Consider a backend 
that does not support APP_COOKIE¹s, or HTTPS_TERMINATED (but has multiple 
listeners) instead.  Should the entire load balancer create fail, or should it 
offer degraded service?  Do all drivers have to implement a transaction 
rollback; wait, the interface makes that very hard.  That¹s his point.  The 
driver is no longer just glue code between interfaces; it¹s now a mini-object 
error handler.


 Having provider defined in multiple places does not make sense.

Channeling Brandon, who can yell if I get this wrong, the point is not to have 
a potentially different provider on each object.  It¹s to allow a provider to 
be assigned when the first object in the tree is created, so that future 
related objects will always get routed to the same provider.
Not knowing which provider should get all the objects is why we have to wait 
until we see a LoadBalancer object.


All of this sort of edge case nonsense is because we (the royal we, the 
community), wanted all load balancer objects to be ³root² objects, even though 
only one of them is an actual root today, to support many-to-many relationships 
among all of them, at some future date, without an interface change.  If my 
bias is showing that I¹m not a fan of adding this complexity for that, I¹m not 
surprised.

Thanks,
doug


On 8/11/14, 7:57 AM, Samuel Bercovici samu...@radware.com wrote:

Hi,
 
Validations such as ³timeout  delay² should be performed on the API 
level before it reaches the driver.
 
For a configuration tree (lb, listeners, pools, etc.), there should be 
one provider.

Having provider defined in multiple places does not make sense.
 
 
-San.
 
 
From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]

Sent: Monday, August 11, 2014 2:43 PM
To: OpenStack Development Mailing List 
(openstack-dev@lists.openstack.org)
Subject: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver 
interface on every API request


 
Hi:
 
Continuing from last week¹s LBaaS meetingŠ
 
Currently an entity cannot be sent to driver unless it is linked to 
loadbalancer because loadbalancer is the root object and driver 
information is only available with loadbalancer.

 
The request to the driver is delayed because of which error propagation 
becomes tricky.
 
Let¹s say a monitor was configured with timeout  delay there would be 
no error then.
When a listener is configured there will be a monitor 
creation/deployment error like ³timeout configured greater than delay².
 
Unless the error is very clearly crafted the user won¹t be able to 
understand the error.
 
I am half-heartedly OK with current approach.

 
But, I would prefer Brandon¹s Solution ­ make provider an attribute in 
each of the entities to get rid of this problem.

 
What do others think?
 
Thanks,
Vijay V.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] devstack local.conf file

2014-08-11 Thread Asselin, Ramy
Hi Nikesh,

You need to set the enabled_backends in the local.conf file.
e.g.

[[post-config|$CINDER_CONF]]
[DEFAULT]
enabled_backends=hp_msa_driver
[hp_msa_driver]
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver
…

Ramy

From: Nikesh Kumar Mahalka [mailto:nikeshmaha...@vedams.com]
Sent: Monday, August 11, 2014 8:55 AM
To: openst...@lists.openstack.org; openstack-dev@lists.openstack.org
Subject: [openstack-dev] devstack local.conf file

Hi,
I have gone through devstack links.
They are not clear like openstack.orghttp://openstack.org documents.


For Example:
when i am using below local.conf file in devstack,hp_msa_driver is not coming 
in enabled_backends in cinder.conf after running stack.sh.

[[local|localrc]]
ADMIN_PASSWORD=vedams123
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=ADMIN
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.2.170/29http://192.168.2.170/29
HOST_IP=192.168.2.151
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
SYSLOG=True
SYSLOG_HOST=$HOST_IP
SYSLOG_PORT=516
TEMPEST_VOLUME_DRIVER=hp_msa_fc
TEMPEST_VOLUME_VENDOR=Hewlett-Packard
TEMPEST_STORAGE_PROTOCOL=FC


[[post-config|$CINDER_CONF]]
[hp_msa_driver]
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver
san_ip = 192.168.2.192
san_login = manage
san_password =!manage
volume_backend_name=HPMSA_FC


[lvmdriver-1]
volume_group=stack-volumes-1
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=LVM_iSCSI



I am getting below cinder.conf file after running stack.sh script

[keystone_authtoken]
auth_uri = http://192.168.2.151:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = vedams123
admin_user = cinder
admin_tenant_name = service
cafile =
identity_uri = http://192.168.2.151:35357

[DEFAULT]
rabbit_password = vedams123
rabbit_hosts = 192.168.2.151
rpc_backend = cinder.openstack.common.rpc.impl_kombu
use_syslog = True
default_volume_type = lvmdriver-1
enabled_backends = lvmdriver-1
enable_v1_api = true
periodic_interval = 60
lock_path = /opt/stack/data/cinder
state_path = /opt/stack/data/cinder
osapi_volume_extension = cinder.api.contrib.standard_extensions
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_config = /etc/cinder/api-paste.ini
sql_connection = 
mysql://root:vedams123@127.0.0.1/cinder?charset=utf8http://root:vedams123@127.0.0.1/cinder?charset=utf8
iscsi_helper = tgtadm
my_ip = 192.168.2.151
verbose = True
debug = True
auth_strategy = keystone

[lvmdriver-1]
volume_group = stack-volumes-1
volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name = LVM_iSCSI

[hp_msa_driver]
volume_backend_name = HPMSA_FC
san_password = !manage
san_login = manage
san_ip = 192.168.2.192
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver



Then i analyzed source code of stack.sh,and added in local.conf this line:
CINDER_ENABLED_BACKENDS=hp_msa:hp_msa_driver,lvm:lvmdriver-1


Now i am getting hp_msa_fc in cinder.conf in enabled_backends



Regards
Nikesh
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Doug Wiegley
Hi Sam,

Very true.  I think that Vijay’s objection is that we are currently
imposing a logical structure on the driver, when it should be a driver
decision.  Certainly, it goes both ways.

And I also agree that the mechanism for returning multiple errors, and the
ability to specify whether those errors are fatal or not, individually, is
currently weak.

Doug


On 8/11/14, 10:21 AM, Samuel Bercovici samu...@radware.com wrote:

Hi Doug,

In some implementations Driver !== Device. I think this is also true for
HA Proxy.
This might mean that there is a difference between creating a logical
object and when there is enough information to actually schedule/place
this into a device.
The ability to express such errors (detecting an error on a logical
object after it was created but when it actually get scheduled) should be
discussed and addressed anyway.

-Sam.


-Original Message-
From: Doug Wiegley [mailto:do...@a10networks.com]
Sent: Monday, August 11, 2014 6:55 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling
driver interface on every API request

Hi all,

 Validations such as ³timeout  delay² should be performed on the API
level before it reaches the driver.
For a configuration tree (lb, listeners, pools, etc.), there should be
one provider.

You¹re right, but I think the point of Vijay¹s example was to highlight
the combo error problem with populating all of the driver objects at once
(in short, the driver interface isn¹t well suited to that model.)  That
his one example can be covered by API validators is irrelevant.  Consider
a backend that does not support APP_COOKIE¹s, or HTTPS_TERMINATED (but
has multiple listeners) instead.  Should the entire load balancer create
fail, or should it offer degraded service?  Do all drivers have to
implement a transaction rollback; wait, the interface makes that very
hard.  That¹s his point.  The driver is no longer just glue code between
interfaces; it¹s now a mini-object error handler.


 Having provider defined in multiple places does not make sense.

Channeling Brandon, who can yell if I get this wrong, the point is not to
have a potentially different provider on each object.  It¹s to allow a
provider to be assigned when the first object in the tree is created, so
that future related objects will always get routed to the same provider.
Not knowing which provider should get all the objects is why we have to
wait until we see a LoadBalancer object.


All of this sort of edge case nonsense is because we (the royal we, the
community), wanted all load balancer objects to be ³root² objects, even
though only one of them is an actual root today, to support many-to-many
relationships among all of them, at some future date, without an
interface change.  If my bias is showing that I¹m not a fan of adding
this complexity for that, I¹m not surprised.

Thanks,
doug


On 8/11/14, 7:57 AM, Samuel Bercovici samu...@radware.com wrote:

Hi,
 
Validations such as ³timeout  delay² should be performed on the API
level before it reaches the driver.
 
For a configuration tree (lb, listeners, pools, etc.), there should be
one provider.

Having provider defined in multiple places does not make sense.
 
 
-San.
 
 
From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]

Sent: Monday, August 11, 2014 2:43 PM
To: OpenStack Development Mailing List
(openstack-dev@lists.openstack.org)
Subject: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver
interface on every API request


 
Hi:
 
Continuing from last week¹s LBaaS meetingŠ
 
Currently an entity cannot be sent to driver unless it is linked to
loadbalancer because loadbalancer is the root object and driver
information is only available with loadbalancer.

 
The request to the driver is delayed because of which error propagation
becomes tricky.
 
Let¹s say a monitor was configured with timeout  delay there would be
no error then.
When a listener is configured there will be a monitor
creation/deployment error like ³timeout configured greater than delay².
 
Unless the error is very clearly crafted the user won¹t be able to
understand the error.
 
I am half-heartedly OK with current approach.

 
But, I would prefer Brandon¹s Solution ­ make provider an attribute in
each of the entities to get rid of this problem.

 
What do others think?
 
Thanks,
Vijay V.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

2014-08-11 Thread Sylvain Bauza


Le 11/08/2014 18:03, Gary Kotton a écrit :


On 8/11/14, 6:06 PM, Dan Smith d...@danplanet.com wrote:


As the person who -2'd the review, I'm thankful you raised this issue on
the ML, Jay. Much appreciated.

The metadetails term isn't being invented in this patch, of course. I
originally complained about the difference when this was being added:

https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contr
ib/server_groups.py,cm

As best I can tell, the response in that patch set about why it's being
translated is wrong (backwards). I expect that the API extension at the
time called it metadetails and they decided to make the object the
same and do the translation there.


From what I can tell, the actual server_group API extension that made it

into the tree never got the ability to set/change/etc the
metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
wrongly.

If we care to have this functionality, then I propose we change the
attribute on the object (we can handle this with versioning) and reflect
it as metadata in the API.

However, I have to ask: do we really need another distinct metadata
store attached to server_groups? If not, how about we just remove it

from the database and the object, clean up the bit of residue that is

still in the API extension and be done with it?

The initial version of the feature did not make use of this. The reason
was that we chose for a very
Limited subset to be used, that is, affinity and anti affinity. Moving
forwards we would like to implement
A number of different policies with this. We can drop it at the moment due
to the fact that it is not used.

I think that Yathi may be using this for the constrain scheduler. But I am
not 100% sure.



Unless I'm wrong, I can't see where this metadata is being used in the 
scheduler, either for filtering or for other reasons.


So, please give us context why this is currently useful ?

If this is something for the next future, I would love discussing it 
with regards to the current split.



Thanks,
-Sylvain


--Dan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree

2014-08-11 Thread Robert Kukura


On 8/8/14, 6:28 PM, Salvatore Orlando wrote:
If we want to keep everything the way it is, we have to change 
everything [1]


This is pretty much how I felt after reading this proposal, and I felt 
that this quote, which Ivar will probably appreciate, was apt to the 
situation.
Recent events have spurred a discussion about the need for a change in 
process. It is not uncommon indeed to believe that by fixing the 
process, things will inevitably change for better. While no-one argues 
that flaws in processes need to be fixed, no process change will ever 
change anything, in my opinion, unless it is aimed at spurring change 
in people as well.


From what I understand, this proposal starts with the assumption that 
any new feature which is committed to Neutron (ie: has a blueprint 
approved), and is not a required neutron component should be 
considered as a preview. This is not different from the process, 
which, albeit more informally, has been adopted so far. Load 
Balancing, Firewall, VPN, have all been explicitly documented as 
experimental in their first release; I would argue that even if not 
experimental anymore, they may not be considered stable until their 
stability was proven by upstream QA with API and scenario tests - but 
this is not sanctioned anywhere currently, I think.
Correct, this proposal is not so much a new process or change in process 
as a formalization of what we've already been doing, and a suggested 
adaptation to clarify the current expectations around stability of new APIs.


According to this proposal, for preview features:
- all the code would be moved to a preview package

Yes.

- Options will be marked as preview

Yes.

- URIs should be prefixed with preview
That's what I suggested, but, as several people have pointed out, this 
does seem like its worth the cost of breaking the API compatibility just 
at the point when it is being declared stable. I'd like to withdraw this 
item.

- CLIs will note the features are preview in their help strings

Yes.
- Documentation will explicitly state this feature is preview (I 
think we already mark them as experimental, frankly I don't think 
there are a lot of differences in terminology here)
Yes. Again to me, failure is one likely outcome of an experiment. The 
term preview is intended to imply more of a commitment to quickly 
reach stability.

- Database migrations will be in the main alembic path as usual

Right.

- CLI, Devstack and Heat support will be available

Right, as appropriate for the feature.

- Can be used by non-preview neutron code
No, I suggested No non-preview Neutron code should import code from 
anywhere under the neutron.preview module, 

- Will undergo the usual review process
Right. This is key for the code to not have to jump through a new major 
upheaval at right as it becomes stable.
- QA will be desirable, but will done either with WIP tempest 
patches or merging the relevant scenario tests in the preview feature 
iself
More than desirable. We need a way to maintain and run the 
tempest-like API and scenario tests during the stabilization process, 
but to let then evolve with the feature.
- The feature might be promoted or removed, but the process for this 
is not yet defined.
Any suggestions? I did try to address preventing long-term stagnation of 
preview features. As a starting point, reviewing and merging a patch 
that moves the code from the preview sub-tree to its intended location 
could be a lightweight promotion process.


I don't think this change in process will actually encourage better 
behaviour both by contributors and core reviewers.
Encouraging better behavior might be necessary, but wasn't the main 
intent of this proposal. This proposal was intended to clarify and 
formalize the stability expectations around the initial releases of new 
features. It was specifically intended to address the conundrum 
currently faced by reviewers regarding patches that meet all applicable 
quality standards, but may not yet have (somehow, miraculously) achieved 
the maturity associated with stable APIs and features fully supported 
for widespread deployment.
I reckon that better behaviour might be encouraged by forcing 
developer and reviewers to merge in the neutron source code tree only 
code which meets the highest quality standards. A change in process 
should enforce this - and when I think about the criteria, I think at 
the same kind of criteria we're being imposed to declare parity with 
nova. Proven reliability, and scalability should be a must. Proven 
usability should be a requirement for all new APIs.
I agree regarding the quality standards for merging of code, and am not 
suggesting relaxing those one bit. But proving all of the desirable 
system-level attributes of a complex new feature before merging anything 
precludes any kind of iterative development process. I think we should 
consider enforcing things like proven reliability, scalability, and 
usability at the point where the 

Re: [openstack-dev] [designate] [neutron] designate and neutron integration

2014-08-11 Thread Carl Baldwin
kazuhiro MIYASHITA,

I have done a lot of thinking about this.  I have a blueprint on hold
until Kilo for Neutron/Designate integration [1].

However, my blueprint doesn't quite address what you are going after
here.  An assumption that I have made is that Designate is an external
or internet facing service so a Neutron router needs to be in the
datapath to carry requests from dnsmasq to an external network.  The
advantage of this is that it is how Neutron works today so there is no
new development needed.

Could you elaborate on the advantages of connecting dnsmasq directly
to the external network where Designate will be available?

Carl

[1] https://review.openstack.org/#/c/88624/

On Mon, Aug 11, 2014 at 7:51 AM, Miyashita, Kazuhiro
miy...@jp.fujitsu.com wrote:
 Hi,

 I want to ask about neutron and designate integration.
 I think dnsmasq fowards DNS request from instance to designate is better.

++
|DNS server(designate)   |
++
 |
 -+--+-- Network1
  |
   ++
   |dnsmasq |
   ++
 |
 -+--+-- Network2
  |
 +-+
 |instance |
 +-+

 Because it's simpler than virtual router connects Network1 and Network2.
 If router connects network, instance should know where DNS server is. it is 
 complicated.
 dnsmasq returns its ip address as dns server in DHCP replay by ordinary, so,
 I think good dnsmasq becomes like a gateway to designate.

 But, I can't connect dnsmasq to Network1. because of today's neutron design.

 Question:
   Does designate design team have a plan such as above integration?
   or other integration design?

 *1: Network1 and Network2 are deployed by neutron.
 *2: neutron deploys dnsmasq as a dhcp server.
 dnsmasq can forward DNS request.

 Thanks,

 kazuhiro MIYASHITA




 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: FW: [Neutron] Group Based Policy and the way forward

2014-08-11 Thread Hemanth Ravi
On Fri, Aug 8, 2014 at 7:13 PM, Armando M. arma...@gmail.com wrote:


 On Fri, Aug 8, 2014 at 5:38 PM, Armando M. arma...@gmail.com wrote:



   One advantage of the service plugin is that one can leverage the
 neutron common framework such as Keystone authentication where common
 scoping is done. It would be important in the policy type of framework to
 have such scoping


 The framework you're referring to is common and already reusable, it's
 not a prerogative of Neutron.


  Are you suggesting that Service Plugins, L3, IPAM etc become individual
 endpoints, resulting in redundant authentication round-trips for each of
 the components.

 Wouldn't this result in degraded performance and potential consistency
 issues?


 The endpoint - in the OpenStack lingo - that exposes the API abstractions
 (concepts and operations) can be, logically and physically, different from
 the worker that implements these abstractions; authentication is orthogonal
 to this and I am not suggesting what you mention.


From what I understand, you are saying that the implementation could be
done via a mechanism different than a service plugin. Would this be done by
implementing the service plugin as a different process? This would imply
making changes to the the neutron server - plugin interface. If this is the
case, wouldn't it be better to use the existing mechanism to avoid
introducing any instability at this stage of the Juno cycle.


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [designate] [neutron] designate and neutron integration

2014-08-11 Thread Hayes, Graham
kazuhiro MIYASHITA,

As designate progresses with server pools, we aim to have support for a
'private' dns server, that could run within a neutron network - is that
the level of integration you were referring to?

That is, for the time being, a long term goal, and not covered by Carl's
Kilo blueprint.

We talked with both people from both Neutron and Nova in Atlanta, and
worked out the first steps for designate / neutron integration (auto
provisioning of records)

For that level of integration, we are assuming that a neutron router
will be involved in DNS queries within a network.

Long term I would prefer to see a 'private pool' connecting directly to
the Network2 (like any other service VM (LBaaS etc)) and have dnsmasq
pass on only records hosted by that 'private pool' to designate.

This is all yet to be fleshed out, so I am open to suggestions. It
requires that we complete server pools, and that work is only just
starting (it was the main focus of our mid-cycle 2 weeks ago).

Graham

On Mon, 2014-08-11 at 11:02 -0600, Carl Baldwin wrote:
 kazuhiro MIYASHITA,
 
 I have done a lot of thinking about this.  I have a blueprint on hold
 until Kilo for Neutron/Designate integration [1].
 
 However, my blueprint doesn't quite address what you are going after
 here.  An assumption that I have made is that Designate is an external
 or internet facing service so a Neutron router needs to be in the
 datapath to carry requests from dnsmasq to an external network.  The
 advantage of this is that it is how Neutron works today so there is no
 new development needed.
 
 Could you elaborate on the advantages of connecting dnsmasq directly
 to the external network where Designate will be available?
 
 Carl
 
 [1] https://review.openstack.org/#/c/88624/
 
 On Mon, Aug 11, 2014 at 7:51 AM, Miyashita, Kazuhiro
 miy...@jp.fujitsu.com wrote:
  Hi,
 
  I want to ask about neutron and designate integration.
  I think dnsmasq fowards DNS request from instance to designate is better.
 
 ++
 |DNS server(designate)   |
 ++
  |
  -+--+-- Network1
   |
++
|dnsmasq |
++
  |
  -+--+-- Network2
   |
  +-+
  |instance |
  +-+
 
  Because it's simpler than virtual router connects Network1 and Network2.
  If router connects network, instance should know where DNS server is. it is 
  complicated.
  dnsmasq returns its ip address as dns server in DHCP replay by ordinary, so,
  I think good dnsmasq becomes like a gateway to designate.
 
  But, I can't connect dnsmasq to Network1. because of today's neutron design.
 
  Question:
Does designate design team have a plan such as above integration?
or other integration design?
 
  *1: Network1 and Network2 are deployed by neutron.
  *2: neutron deploys dnsmasq as a dhcp server.
  dnsmasq can forward DNS request.
 
  Thanks,
 
  kazuhiro MIYASHITA
 
 
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] oslo.concurrency repo review

2014-08-11 Thread Yuriy Taraday
On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow harlo...@outlook.com wrote:

 One question from me:

 Will there be later fixes to remove oslo.config dependency/usage from
 oslo.concurrency?

 I still don't understand how oslo.concurrency can be used as a library
 with the configuration being set in a static manner via oslo.config (let's
 use the example of `lock_path` @ https://github.com/YorikSar/
 oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). For
 example:

 Library X inside application Z uses lockutils (via the nice
 oslo.concurrency library) and sets the configuration `lock_path` to its
 desired settings, then library Y (also a user of oslo.concurrency) inside
 same application Z sets the configuration for `lock_path` to its desired
 settings. Now both have some unknown set of configuration they have set and
 when library X (or Y) continues to use lockutils they will be using some
 mix of configuration (likely some mish mash of settings set by X and Y);
 perhaps to a `lock_path` that neither actually wants to be able to write
 to...

 This doesn't seem like it will end well; and will just cause headaches
 during debug sessions, testing, integration and more...

 The same question can be asked about the `set_defaults()` function, how is
 library Y or X expected to use this (are they?)??

 I hope one of the later changes is to remove/fix this??

 Thoughts?

 -Josh


I'd be happy to remove lock_path config variable altogether. It's basically
never used. There are two basic branches in code wrt lock_path:
- when you provide lock_path argument to lock (and derivative functions),
file-based lock is used and CONF.lock_path is ignored;
- when you don't provide lock_path in arguments, semaphore-based lock is
used and CONF.lock_path is just a prefix for its name (before hashing).

I wonder if users even set lock_path in their configs as it has almost no
effect. So I'm all for removing it, but...
From what I understand, every major change in lockutils drags along a lot
of headache for everybody (and risk of bugs that would be discovered very
late). So is such change really worth it? And if so, it will require very
thorough research of lockutils usage patterns.

-- 

Kind regards, Yuriy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Which program for Rally

2014-08-11 Thread Zane Bitter

On 08/08/14 10:41, Anne Gentle wrote:

- Would have to ensure Rally is what we want first as getting to be PTL
since you are first to propose seems to be the model.


I know that at one time it was popular in the trade/gutter press to cast 
aspersions on new projects by saying that someone getting to be a PTL 
was the major motivation behind them. And although, having been there, I 
can tell you that this was grossly unfair to the people concerned, at 
least you could see where the impression might have come from in the 
days where being a PTL guaranteed you a seat on the TC.


These days with a directly elected TC, the job of a PTL is confined to 
administrative busywork. To the extent that a PTL holds any real ex 
officio power, which is not a great extent, it's probably a mistake that 
will soon be rectified. If anyone is really motivated to become a PTL by 
their dreams of avarice then I can guarantee that they will be disappointed.


It seems pretty clear to me that projects want their own programs 
because they don't think it wise to hand over control of all changes to 
the thing they've been working on for the past year or more to a group 
of people who have barely glanced at it before and already have other 
priorities. I submit that this is sufficient to completely explain the 
proliferation of programs without attributing to anyone any untoward 
motivations.


Finally, *yes*, the model is indeed that the project working in the open 
with the community eventually gets incubated, and the proprietary 
project working behind closed doors with a plan to 'open source' it one 
day, when it's perfect is doomed to perpetual irrelevance. You'll note 
that anyone who is unhappy about that still has an obvious course of 
action that doesn't involve punishing the people who are trying to do 
the Right Thing by the community.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Brad Topol
Hi Eoghan,

Thanks for the note below.  However, one thing the overview below does not 
 cover  is why InfluxDB (http://influxdb.com/) is not being leveraged. 
Many folks feel that this technology is a viable solution for the problem 
space discussed below.

Thanks,

Brad


Brad Topol, Ph.D.
IBM Distinguished Engineer
OpenStack
(919) 543-0646
Internet:  bto...@us.ibm.com
Assistant: Kendra Witherspoon (919) 254-0680



From:   Eoghan Glynn egl...@redhat.com
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.org, 
Date:   08/06/2014 11:17 AM
Subject:[openstack-dev] [tc][ceilometer] Some background on the 
gnocchi project




Folks,

It's come to our attention that some key individuals are not
fully up-to-date on gnocchi activities, so it being a good and
healthy thing to ensure we're as communicative as possible about
our roadmap, I've provided a high-level overview here of our
thinking. This is intended as a precursor to further discussion
with the TC.

Cheers,
Eoghan


What gnocchi is:
===

Gnocchi is a separate, but related, project spun up on stackforge
by Julien Danjou, with the objective of providing efficient
storage and retrieval of timeseries-oriented data and resource
representations.

The goal is to experiment with a potential approach to addressing
an architectural misstep made in the very earliest days of
ceilometer, specifically the decision to store snapshots of some
resource metadata alongside each metric datapoint. The core idea
is to move to storing datapoints shorn of metadata, and instead
allow the resource-state timeline to be reconstructed more cheaply
from much less frequently occurring events (e.g. instance resizes
or migrations).


What gnocchi isn't:
==

Gnocchi is not a large-scale under-the-radar rewrite of a core
OpenStack component along the lines of keystone-lite.

The change is concentrated on the final data-storage phase of
the ceilometer pipeline, so will have little initial impact on the
data-acquiring agents, or on transformation phase.

We've been totally open at the Atlanta summit and other forums
about this approach being a multi-cycle effort.


Why we decided to do it this way:


The intent behind spinning up a separate project on stackforge
was to allow the work progress at arms-length from ceilometer,
allowing normalcy to be maintained on the core project and a
rapid rate of innovation on gnocchi.

Note that that the developers primarily contributing to gnocchi
represent a cross-section of the core team, and there's a regular
feedback loop in the form of a recurring agenda item at the
weekly team meeting to avoid the effort becoming silo'd.


But isn't re-architecting frowned upon?
==

Well, the architecture of other OpenStack projects have also
under-gone change as the community understanding of the
implications of prior design decisions has evolved.

Take for example the move towards nova no-db-compute  the
unified-object-model in order to address issues in the nova
architecture that made progress towards rolling upgrades
unneccessarily difficult.

The point, in my understanding, is not to avoid doing the
course-correction where it's deemed necessary. Rather, the
principle is more that these corrections happen in an open
and planned way.


The path forward:


A subset of the ceilometer community will continue to work on
gnocchi in parallel with the ceilometer core over the remainder
of the Juno cycle and into the Kilo timeframe. The goal is to
have an initial implementation of gnocchi ready for tech preview
by the end of Juno, and to have the integration/migration/
co-existence questions addressed in Kilo.

Moving the ceilometer core to using gnocchi will be contingent
on it demonstrating the required performance characteristics and
providing the semantics needed to support a v3 ceilometer API
that's fit-for-purpose.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

2014-08-11 Thread Clint Byrum
Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:
 On 11/08/14 10:46, Clint Byrum wrote:
  Right now we're stuck with an update that just doesn't work. It isn't
  just about update-failure-recovery, which is coming along nicely, but
  it is also about the lack of signals to control rebuild, poor support
  for addressing machines as groups, and unacceptable performance in
  large stacks.
 
 Are there blueprints/bugs filed for all of these issues?
 

Convergnce addresses the poor performance for large stacks in general.
We also have this:

https://bugs.launchpad.net/heat/+bug/1306743

Which shows how slow metadata access can get. I have worked on patches
but haven't been able to complete them. We made big strides but we are
at a point where 40 nodes polling Heat every 30s is too much for one CPU
to handle. When we scaled Heat out onto more CPUs on one box by forking
we ran into eventlet issues. We also ran into issues because even with
many processes we can only use one to resolve templates for a single
stack during update, which was also excessively slow.

We haven't been able to come back around to those yet, but you can see
where this has turned into a bit of a rat hole of optimization.

action-aware-sw-config is sort of what we want for rebuild. We
collaborated with the trove devs on how to also address it for resize
a while back but I have lost track of that work as it has taken a back
seat to more pressing issues.

Addressing groups is a general problem that I've had a hard time
articulating in the past. Tomas Sedovic has done a good job with this
TripleO spec, but I don't know that we've asked for an explicit change
in a bug or spec in Heat just yet:

https://review.openstack.org/#/c/97939/

There are a number of other issues noted in that spec which are already
addressed in Heat, but require refactoring in TripleO's templates and
tools, and that work continues.

The point remains: we need something that works now, and doing an
alternate implementation for updates is actually faster than addressing
all of these issues.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [operstack-dev][Congress] congress-server fails

2014-08-11 Thread Rajdeep Dua
Following command gives an error

./*bin*/*congress*-*server* --config-file etc/congress.conf.sample

Error i am getting is ImportError: No module named
keystonemiddleware.auth_token

Tried installing keystonemiddleware manually, got the following error

$ sudo pip install keystonemiddleware
Downloading/unpacking keystonemiddleware
  Running setup.py egg_info for package keystonemiddleware
[pbr] Reusing existing SOURCES.txt
Requirement already satisfied (use --upgrade to upgrade): Babel=1.3 in
/usr/local/lib/python2.7/dist-packages (from keystonemiddleware)
Requirement already satisfied (use --upgrade to upgrade): iso8601=0.1.9 in
/usr/local/lib/python2.7/dist-packages (from keystonemiddleware)
Requirement already satisfied (use --upgrade to upgrade): netaddr=0.7.6 in
/usr/local/lib/python2.7/dist-packages (from keystonemiddleware)
Downloading/unpacking oslo.config=1.4.0.0a3 (from keystonemiddleware)
  Could not find a version that satisfies the requirement
oslo.config=1.4.0.0a3 (from keystonemiddleware) (from versions: )
No distributions matching the version for oslo.config=1.4.0.0a3 (from
keystonemiddleware)
Storing complete log in /home/hadoop/.pip/pip.log

Thanks
Rajdeep
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-11 Thread Jay Pipes

Hi Li, comments inline.

On 08/08/2014 12:03 AM, Li Ma wrote:

Getting a massive amount of information from data storage to be displayed is
where most of the activity happens in OpenStack. The two activities of reading
data and writing (creating, updating and deleting) data are fundamentally
different.

The optimization for these two opposite database activities can be done by
physically separating the databases that service these two different
activities. All the writes go to database servers, which then replicates the
written data to the database server(s) dedicated to servicing the reads.

Currently, AFAIK, many OpenStack deployment in production try to take
advantage of MySQL (includes Percona or MariaDB) multi-master Galera cluster.
It is possible to design and implement a read/write separation schema
for such a DB cluster.


The above does not really make sense for MySQL Galera/PXC clusters *if 
only Galera nodes are used in the cluster*. Since Galera is 
synchronously replicated, there's no real point in segregating writers 
from readers, IMO. Better to just spread the write AND read load equally 
among all Galera cluster nodes.


However, if you have a Galera cluster that then slaves off to one or 
more standard MySQL slaves, then certainly doing writer/reader 
segregation could be useful, especially for directing readers of 
aggregate or report-type data to the read-only slaves.



Actually, OpenStack has a method for read scalability via defining
master_connection and slave_connection in configuration, but this method
lacks of flexibility due to deciding master or slave in the logical
context(code). It's not transparent for application developer.
As a result, it is not widely used in all the OpenStack projects.

So, I'd like to propose a transparent read/write separation method
for oslo.db that every project may happily takes advantage of it
without any code modification.


I've never seen a writer/reader segregation proxy or middleware piece 
that was properly able to send the right reads to the slaves. 
Unfortunately, determining what are the right reads to send to the 
slaves is highly application-dependent, since the application knows when 
it can tolerate slave lags.



Moreover, I'd like to put it in the mailing list in advance to
make sure it is acceptable for oslo.db.


I think oslo.db is not the right place for this. I believe the efforts 
that Mike Wilson has been doing in the slavification blueprints are 
the more appropriate place to add this slave-aware code.


Best,
-jay


I'd appreciate any comments.

br.
Li Ma


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Congress] congress-server fails to start

2014-08-11 Thread Rajdeep Dua
Hi All,
command to start the congress-server fails

$ ./bin/congress-server --config-file etc/congress.conf.sample

Error :
ImportError: No module named keystonemiddleware.auth_token

Installing keystonemiddleware manually also fails

$ sudo pip install keystonemiddleware

Could not find a version that satisfies the requirement
oslo.config=1.4.0.0a3 (from keystonemiddleware) (from versions: )
No distributions matching the version for oslo.config=1.4.0.0a3 (from
keystonemiddleware)

Thanks
Rajdeep
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

2014-08-11 Thread Steven Hardy
On Mon, Aug 11, 2014 at 11:20:50AM -0700, Clint Byrum wrote:
 Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:
  On 11/08/14 10:46, Clint Byrum wrote:
   Right now we're stuck with an update that just doesn't work. It isn't
   just about update-failure-recovery, which is coming along nicely, but
   it is also about the lack of signals to control rebuild, poor support
   for addressing machines as groups, and unacceptable performance in
   large stacks.
  
  Are there blueprints/bugs filed for all of these issues?
  
 
 Convergnce addresses the poor performance for large stacks in general.
 We also have this:
 
 https://bugs.launchpad.net/heat/+bug/1306743
 
 Which shows how slow metadata access can get. I have worked on patches
 but haven't been able to complete them. We made big strides but we are
 at a point where 40 nodes polling Heat every 30s is too much for one CPU
 to handle. When we scaled Heat out onto more CPUs on one box by forking
 we ran into eventlet issues. We also ran into issues because even with
 many processes we can only use one to resolve templates for a single
 stack during update, which was also excessively slow.

Related to this, and a discussion we had recently at the TripleO meetup is
this spec I raised today:

https://review.openstack.org/#/c/113296/

It's following up on the idea that we could potentially address (or at
least mitigate, pending the fully convergence-ified heat) some of these
scalability concerns, if TripleO moves from the one-giant-template model
to a more modular nested-stack/provider model (e.g what Tomas has been
working on)

I've not got into enough detail on that yet to be sure if it's acheivable
for Juno, but it seems initially to be complex-but-doable.

I'd welcome feedback on that idea and how it may fit in with the more
granular convergence-engine model.

Can you link to the eventlet/forking issues bug please?  I thought since
bug #1321303 was fixed that multiple engines and multiple workers should
work OK, and obviously that being true is a precondition to expending
significant effort on the nested stack decoupling plan above.

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] oslo.concurrency repo review

2014-08-11 Thread Ben Nemec
On 08/11/2014 01:02 PM, Yuriy Taraday wrote:
 On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow harlo...@outlook.com wrote:
 
 One question from me:

 Will there be later fixes to remove oslo.config dependency/usage from
 oslo.concurrency?

 I still don't understand how oslo.concurrency can be used as a library
 with the configuration being set in a static manner via oslo.config (let's
 use the example of `lock_path` @ https://github.com/YorikSar/
 oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). For
 example:

 Library X inside application Z uses lockutils (via the nice
 oslo.concurrency library) and sets the configuration `lock_path` to its
 desired settings, then library Y (also a user of oslo.concurrency) inside
 same application Z sets the configuration for `lock_path` to its desired
 settings. Now both have some unknown set of configuration they have set and
 when library X (or Y) continues to use lockutils they will be using some
 mix of configuration (likely some mish mash of settings set by X and Y);
 perhaps to a `lock_path` that neither actually wants to be able to write
 to...

 This doesn't seem like it will end well; and will just cause headaches
 during debug sessions, testing, integration and more...

 The same question can be asked about the `set_defaults()` function, how is
 library Y or X expected to use this (are they?)??

 I hope one of the later changes is to remove/fix this??

 Thoughts?

 -Josh
 
 
 I'd be happy to remove lock_path config variable altogether. It's basically
 never used. There are two basic branches in code wrt lock_path:
 - when you provide lock_path argument to lock (and derivative functions),
 file-based lock is used and CONF.lock_path is ignored;
 - when you don't provide lock_path in arguments, semaphore-based lock is
 used and CONF.lock_path is just a prefix for its name (before hashing).
 
 I wonder if users even set lock_path in their configs as it has almost no
 effect. So I'm all for removing it, but...
 From what I understand, every major change in lockutils drags along a lot
 of headache for everybody (and risk of bugs that would be discovered very
 late). So is such change really worth it? And if so, it will require very
 thorough research of lockutils usage patterns.

Two things lock_path has to stay for: Windows and consumers who require
file-based locking semantics.  Neither of those use cases are trivial to
remove, so IMHO it would not be appropriate to do it as part of the
graduation.  If we were going to alter the API that much it needed to
happen in incubator.

As far as lock_path mismatches, that shouldn't be a problem unless a
consumer is doing something very unwise.  Oslo libs get their
configuration from the application using them, so unless the application
passes two separate conf objects to library X and Y they're both going
to get consistent settings.  If someone _is_ doing that, then I think
it's their responsibility to make sure the options in both config files
are compatible with each other.

-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Infra] Meeting Tuesday August 12th at 19:00 UTC

2014-08-11 Thread Elizabeth K. Joseph
Hi everyone,

The OpenStack Infrastructure (Infra) team is hosting our weekly
meeting on Tuesday August 12th, at 19:00 UTC in #openstack-meeting

Meeting agenda available here:
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting (anyone is
welcome to to add agenda items)

Everyone interested in infrastructure and process surrounding
automated testing and deployment is encouraged to attend.

-- 
Elizabeth Krumbach Joseph || Lyz || pleia2
http://www.princessleia.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

2014-08-11 Thread Clint Byrum
Excerpts from Steven Hardy's message of 2014-08-11 11:40:07 -0700:
 On Mon, Aug 11, 2014 at 11:20:50AM -0700, Clint Byrum wrote:
  Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:
   On 11/08/14 10:46, Clint Byrum wrote:
Right now we're stuck with an update that just doesn't work. It isn't
just about update-failure-recovery, which is coming along nicely, but
it is also about the lack of signals to control rebuild, poor support
for addressing machines as groups, and unacceptable performance in
large stacks.
   
   Are there blueprints/bugs filed for all of these issues?
   
  
  Convergnce addresses the poor performance for large stacks in general.
  We also have this:
  
  https://bugs.launchpad.net/heat/+bug/1306743
  
  Which shows how slow metadata access can get. I have worked on patches
  but haven't been able to complete them. We made big strides but we are
  at a point where 40 nodes polling Heat every 30s is too much for one CPU
  to handle. When we scaled Heat out onto more CPUs on one box by forking
  we ran into eventlet issues. We also ran into issues because even with
  many processes we can only use one to resolve templates for a single
  stack during update, which was also excessively slow.
 
 Related to this, and a discussion we had recently at the TripleO meetup is
 this spec I raised today:
 
 https://review.openstack.org/#/c/113296/
 
 It's following up on the idea that we could potentially address (or at
 least mitigate, pending the fully convergence-ified heat) some of these
 scalability concerns, if TripleO moves from the one-giant-template model
 to a more modular nested-stack/provider model (e.g what Tomas has been
 working on)
 
 I've not got into enough detail on that yet to be sure if it's acheivable
 for Juno, but it seems initially to be complex-but-doable.
 
 I'd welcome feedback on that idea and how it may fit in with the more
 granular convergence-engine model.
 
 Can you link to the eventlet/forking issues bug please?  I thought since
 bug #1321303 was fixed that multiple engines and multiple workers should
 work OK, and obviously that being true is a precondition to expending
 significant effort on the nested stack decoupling plan above.
 

That was the issue. So we fixed that bug, but we never un-reverted
the patch that forks enough engines to use up all the CPU's on a box
by default. That would likely help a lot with metadata access speed
(we could manually do it in TripleO but we tend to push defaults. :)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [TripleO] reverting the HOT migration? // dealing with lockstep changes

2014-08-11 Thread Robert Collins
Hi, so shortly after the HOT migration landed, we hit
https://bugs.launchpad.net/tripleo/+bug/1354305 which is that on even
quite recently deployed clouds, the migrated templates were just too
new. A partial revert (of just the list_join bit) fixes that, but a
deeper problem emerged which is that stack-update to get from a
non-HOT to HOT template appears broken
(https://bugs.launchpad.net/heat/+bug/1354962).

I think we need to revert the HOT migration today, as forcing a
scorched earth recreation of a cloud is not a great answer for folk
that have deployed versions - its a backwards compat issue.

Its true that our release as of icehouse isn't  really useable, so we
could try to wiggle our way past this one, but I think as the first
real test of our new backwards compat policy, that that would be a
mistake.

What we need to be able to land it again, is some way whereby an
existing cloud can upgrade their undercloud (via stack-update against
the old heat deployed in the seed [today, old heat in the undercloud
itself in future]) and then once that is deployed subsequent templates
can use the new features. We're likely to run into such lockstep
changes in future, so we also need to be able to recognise them in
review / design, and call them out so we can fix them early rather
than deep down the pike.

-Rob

-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] usage patterns for oslo.config

2014-08-11 Thread Doug Hellmann

On Aug 8, 2014, at 7:22 PM, Devananda van der Veen devananda@gmail.com 
wrote:

 On Fri, Aug 8, 2014 at 12:41 PM, Doug Hellmann d...@doughellmann.com wrote:
 
 That’s right. The preferred approach is to put the register_opt() in
 *runtime* code somewhere before the option will be used. That might be in
 the constructor for a class that uses an option, for example, as described
 in
 http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options
 
 Doug
 
 Interesting.
 
 I've been following the prevailing example in Nova, which is to
 register opts at the top of a module, immediately after defining them.
 Is there a situation in which one approach is better than the other?

The approach used in Nova is the “old” way of doing it. It works, but assumes 
that all of the application code is modifying a global configuration object. 
The runtime approach allows you to pass a configuration object to a library, 
which makes it easier to mock the configuration for testing and avoids having 
the configuration options bleed into the public API of the library. We’ve 
started using the runtime approach in new Oslo libraries that have 
configuration options, but changing the implementation in existing application 
code isn’t strictly necessary.

Doug

 
 Thanks,
 Devananda
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Eoghan Glynn


 Hi Eoghan,
 
 Thanks for the note below. However, one thing the overview below does not
 cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged. Many
 folks feel that this technology is a viable solution for the problem space
 discussed below.

Great question Brad!

As it happens we've been working closely with Paul Dix (lead
developer of InfluxDB) to ensure that this metrics store would be
usable as a backend driver. That conversation actually kicked off
at the Juno summit in Atlanta, but it really got off the ground
at our mid-cycle meet-up in Paris on in early July.

I wrote a rough strawman version of an InfluxDB driver in advance
of the mid-cycle to frame the discussion, and Paul Dix traveled
to the meet-up so we could have the discussion face-to-face. The
conclusion was that InfluxDB would indeed potentially be a great
fit, modulo some requirements that we identified during the detailed
discussions:

 * shard-space-based retention  backgrounded deletion
 * capability to merge individual timeseries for cross-aggregation
 * backfill-aware downsampling

The InfluxDB folks have committed to implementing those features in
over July and August, and have made concrete progress on that score.

I hope that provides enough detail to answer to your question?

Cheers,
Eoghan

 Thanks,
 
 Brad
 
 
 Brad Topol, Ph.D.
 IBM Distinguished Engineer
 OpenStack
 (919) 543-0646
 Internet: bto...@us.ibm.com
 Assistant: Kendra Witherspoon (919) 254-0680
 
 
 
 From: Eoghan Glynn egl...@redhat.com
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org,
 Date: 08/06/2014 11:17 AM
 Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
 project
 
 
 
 
 
 Folks,
 
 It's come to our attention that some key individuals are not
 fully up-to-date on gnocchi activities, so it being a good and
 healthy thing to ensure we're as communicative as possible about
 our roadmap, I've provided a high-level overview here of our
 thinking. This is intended as a precursor to further discussion
 with the TC.
 
 Cheers,
 Eoghan
 
 
 What gnocchi is:
 ===
 
 Gnocchi is a separate, but related, project spun up on stackforge
 by Julien Danjou, with the objective of providing efficient
 storage and retrieval of timeseries-oriented data and resource
 representations.
 
 The goal is to experiment with a potential approach to addressing
 an architectural misstep made in the very earliest days of
 ceilometer, specifically the decision to store snapshots of some
 resource metadata alongside each metric datapoint. The core idea
 is to move to storing datapoints shorn of metadata, and instead
 allow the resource-state timeline to be reconstructed more cheaply
 from much less frequently occurring events (e.g. instance resizes
 or migrations).
 
 
 What gnocchi isn't:
 ==
 
 Gnocchi is not a large-scale under-the-radar rewrite of a core
 OpenStack component along the lines of keystone-lite.
 
 The change is concentrated on the final data-storage phase of
 the ceilometer pipeline, so will have little initial impact on the
 data-acquiring agents, or on transformation phase.
 
 We've been totally open at the Atlanta summit and other forums
 about this approach being a multi-cycle effort.
 
 
 Why we decided to do it this way:
 
 
 The intent behind spinning up a separate project on stackforge
 was to allow the work progress at arms-length from ceilometer,
 allowing normalcy to be maintained on the core project and a
 rapid rate of innovation on gnocchi.
 
 Note that that the developers primarily contributing to gnocchi
 represent a cross-section of the core team, and there's a regular
 feedback loop in the form of a recurring agenda item at the
 weekly team meeting to avoid the effort becoming silo'd.
 
 
 But isn't re-architecting frowned upon?
 ==
 
 Well, the architecture of other OpenStack projects have also
 under-gone change as the community understanding of the
 implications of prior design decisions has evolved.
 
 Take for example the move towards nova no-db-compute  the
 unified-object-model in order to address issues in the nova
 architecture that made progress towards rolling upgrades
 unneccessarily difficult.
 
 The point, in my understanding, is not to avoid doing the
 course-correction where it's deemed necessary. Rather, the
 principle is more that these corrections happen in an open
 and planned way.
 
 
 The path forward:
 
 
 A subset of the ceilometer community will continue to work on
 gnocchi in parallel with the ceilometer core over the remainder
 of the Juno cycle and into the Kilo timeframe. The goal is to
 have an initial implementation of gnocchi ready for tech preview
 by the end of Juno, and to have the integration/migration/
 co-existence questions addressed in Kilo.
 
 Moving the ceilometer core to using gnocchi will be contingent
 on it 

Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Vijay Venkatachalam

Yes, the point was to say the plugin need not restrict and let driver decide 
what to do with the API.

Even if the call was made to driver instantaneously, I understand, the driver 
might decide to ignore 
first and schedule later. But, if the call is present, there is scope for 
validation. 
Also, the driver might be scheduling an async-api to backend, in which case  
deployment error 
cannot be shown to the user instantaneously.

W.r.t. identifying a provider/driver, how would it be to make tenant the 
default root object? 
tenantid is already associated with each of these entities, so no additional 
pain. 
For the tenant who wants to override let him specify provider in each of the 
entities.
If you think of this in terms of the UI, let's say if the loadbalancer 
configuration is exposed 
as a single wizard (which has loadbalancer, listener, pool, monitor properties) 
then provider
 is chosen only once. 

Curious question, is flavour framework expected to address this problem?

Thanks,
Vijay V.

-Original Message-
From: Doug Wiegley [mailto:do...@a10networks.com] 
Sent: 11 August 2014 22:02
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver 
interface on every API request

Hi Sam,

Very true.  I think that Vijay’s objection is that we are currently imposing a 
logical structure on the driver, when it should be a driver decision.  
Certainly, it goes both ways.

And I also agree that the mechanism for returning multiple errors, and the 
ability to specify whether those errors are fatal or not, individually, is 
currently weak.

Doug


On 8/11/14, 10:21 AM, Samuel Bercovici samu...@radware.com wrote:

Hi Doug,

In some implementations Driver !== Device. I think this is also true 
for HA Proxy.
This might mean that there is a difference between creating a logical 
object and when there is enough information to actually schedule/place 
this into a device.
The ability to express such errors (detecting an error on a logical 
object after it was created but when it actually get scheduled) should 
be discussed and addressed anyway.

-Sam.


-Original Message-
From: Doug Wiegley [mailto:do...@a10networks.com]
Sent: Monday, August 11, 2014 6:55 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling 
driver interface on every API request

Hi all,

 Validations such as ³timeout  delay² should be performed on the API 
level before it reaches the driver.
For a configuration tree (lb, listeners, pools, etc.), there should be 
one provider.

You¹re right, but I think the point of Vijay¹s example was to highlight 
the combo error problem with populating all of the driver objects at 
once (in short, the driver interface isn¹t well suited to that model.)  
That his one example can be covered by API validators is irrelevant.  
Consider a backend that does not support APP_COOKIE¹s, or 
HTTPS_TERMINATED (but has multiple listeners) instead.  Should the 
entire load balancer create fail, or should it offer degraded service?  
Do all drivers have to implement a transaction rollback; wait, the 
interface makes that very hard.  That¹s his point.  The driver is no 
longer just glue code between interfaces; it¹s now a mini-object error handler.


 Having provider defined in multiple places does not make sense.

Channeling Brandon, who can yell if I get this wrong, the point is not 
to have a potentially different provider on each object.  It¹s to allow 
a provider to be assigned when the first object in the tree is created, 
so that future related objects will always get routed to the same provider.
Not knowing which provider should get all the objects is why we have to 
wait until we see a LoadBalancer object.


All of this sort of edge case nonsense is because we (the royal we, the 
community), wanted all load balancer objects to be ³root² objects, even 
though only one of them is an actual root today, to support 
many-to-many relationships among all of them, at some future date, 
without an interface change.  If my bias is showing that I¹m not a fan 
of adding this complexity for that, I¹m not surprised.

Thanks,
doug


On 8/11/14, 7:57 AM, Samuel Bercovici samu...@radware.com wrote:

Hi,
 
Validations such as ³timeout  delay² should be performed on the API 
level before it reaches the driver.
 
For a configuration tree (lb, listeners, pools, etc.), there should be 
one provider.

Having provider defined in multiple places does not make sense.
 
 
-San.
 
 
From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]

Sent: Monday, August 11, 2014 2:43 PM
To: OpenStack Development Mailing List
(openstack-dev@lists.openstack.org)
Subject: [openstack-dev] [Neutron][LBaaS] Continuing on Calling 
driver interface on every API request


 
Hi:
 
Continuing from last week¹s LBaaS meetingŠ
 
Currently an entity cannot be sent to driver unless it 

Re: [openstack-dev] [oslo] oslo.concurrency repo review

2014-08-11 Thread Joshua Harlow



On Mon, Aug 11, 2014 at 11:39 AM, Ben Nemec openst...@nemebean.com 
wrote:

On 08/11/2014 01:02 PM, Yuriy Taraday wrote:
 On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow 
harlo...@outlook.com wrote:
 

 One question from me:

 Will there be later fixes to remove oslo.config dependency/usage 
from

 oslo.concurrency?

 I still don't understand how oslo.concurrency can be used as a 
library
 with the configuration being set in a static manner via 
oslo.config (let's

 use the example of `lock_path` @ https://github.com/YorikSar/
 oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
For

 example:

 Library X inside application Z uses lockutils (via the nice
 oslo.concurrency library) and sets the configuration `lock_path` 
to its
 desired settings, then library Y (also a user of oslo.concurrency) 
inside
 same application Z sets the configuration for `lock_path` to its 
desired
 settings. Now both have some unknown set of configuration they 
have set and
 when library X (or Y) continues to use lockutils they will be 
using some
 mix of configuration (likely some mish mash of settings set by X 
and Y);
 perhaps to a `lock_path` that neither actually wants to be able to 
write

 to...

 This doesn't seem like it will end well; and will just cause 
headaches

 during debug sessions, testing, integration and more...

 The same question can be asked about the `set_defaults()` 
function, how is

 library Y or X expected to use this (are they?)??

 I hope one of the later changes is to remove/fix this??

 Thoughts?

 -Josh
 
 
 I'd be happy to remove lock_path config variable altogether. It's 
basically

 never used. There are two basic branches in code wrt lock_path:
 - when you provide lock_path argument to lock (and derivative 
functions),

 file-based lock is used and CONF.lock_path is ignored;
 - when you don't provide lock_path in arguments, semaphore-based 
lock is
 used and CONF.lock_path is just a prefix for its name (before 
hashing).
 
 I wonder if users even set lock_path in their configs as it has 
almost no

 effect. So I'm all for removing it, but...
 From what I understand, every major change in lockutils drags along 
a lot
 of headache for everybody (and risk of bugs that would be 
discovered very
 late). So is such change really worth it? And if so, it will 
require very

 thorough research of lockutils usage patterns.


Two things lock_path has to stay for: Windows and consumers who 
require
file-based locking semantics.  Neither of those use cases are trivial 
to

remove, so IMHO it would not be appropriate to do it as part of the
graduation.  If we were going to alter the API that much it needed to
happen in incubator.


As far as lock_path mismatches, that shouldn't be a problem unless a
consumer is doing something very unwise.  Oslo libs get their
configuration from the application using them, so unless the 
application

passes two separate conf objects to library X and Y they're both going
to get consistent settings.  If someone _is_ doing that, then I think
it's their responsibility to make sure the options in both config 
files

are compatible with each other.


Why would it be assumed they would pass the same settings (how is that 
even possible to know ahead of time? especially if library X pulls in a 
new library ZZ that requires a new configuration setting). For example, 
one directory for `lock_path` may be reasonable for tooz and another 
may be reasonable for taskflow (completly depends on there intended 
usage), it would not likely desireable to have them go to the same 
location. Forcing application Z to know the inner workings of library X 
and library Y (or future unknown library ZZ) is just pushing the 
problem onto the library user, which seems inappropriate and breaks the 
whole point of having abstractions  APIs in the first place... This 
IMHO is part of the problem with having statically set *action at a 
distance* type of configuration, the libraries themselves are not in 
control of their own configuration, which breaks abstractions  APIs 
left and right. If some application Y can go under a library and pull 
the rug out from under it, how is that a reasonable thing to expect the 
library to be able to predict  handle?


This kind of requirement has always made me wonder how other libraries 
(like tooz, or taskflow) actually interact with any of the oslo.* 
libraries in any predicatable way (since those library could be 
interacting with oslo.* libraries that have configuration that can be 
switched out from underneath them, making those libraries have *secret* 
APIs that appear and disappear depending on what used oslo.* library 
was newly added as a dependency and what newly added configuration that 
library sucked in/exposed...).


-Josh




-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




Re: [openstack-dev] [TripleO] reverting the HOT migration? // dealing with lockstep changes

2014-08-11 Thread Dan Prince
On Tue, 2014-08-12 at 06:58 +1200, Robert Collins wrote:
 Hi, so shortly after the HOT migration landed, we hit
 https://bugs.launchpad.net/tripleo/+bug/1354305 which is that on even
 quite recently deployed clouds, the migrated templates were just too
 new. A partial revert (of just the list_join bit) fixes that, but a
 deeper problem emerged which is that stack-update to get from a
 non-HOT to HOT template appears broken
 (https://bugs.launchpad.net/heat/+bug/1354962).
 
 I think we need to revert the HOT migration today, as forcing a
 scorched earth recreation of a cloud is not a great answer for folk
 that have deployed versions - its a backwards compat issue.
 
 Its true that our release as of icehouse isn't  really useable, so we
 could try to wiggle our way past this one, but I think as the first
 real test of our new backwards compat policy, that that would be a
 mistake.

Hmmm. We blocked a good bit of changes to get these HOT templates in so
I hate to see us revert them. Also, It isn't clear to me how much work
it would be to fully support the non-HOT to HOT templates upgrade path.
How much work is this? And is that something we really want to spend
time on instead of all the other things?

Why not just create a branch of the old templates that works on the
existing underclouds? Users would then need to use these templates as a
one-time upgrade step to be able to upgrade to a heat HOT capable
undercloud first.

With regards to the seed...

Would a tool that allowed us to migrate the stateful data from an old
seed to a new seed be a better use of time here? A re-seeder might be a
useful tool to future proof upgrade paths that rely on the software
versions in the seed (Heat etc.) that aren't easily up-gradable.
Packages would help here too, but not everyone uses them...

Dan

 
 What we need to be able to land it again, is some way whereby an
 existing cloud can upgrade their undercloud (via stack-update against
 the old heat deployed in the seed [today, old heat in the undercloud
 itself in future]) and then once that is deployed subsequent templates
 can use the new features. We're likely to run into such lockstep
 changes in future, so we also need to be able to recognise them in
 review / design, and call them out so we can fix them early rather
 than deep down the pike.
 
 -Rob
 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] oslo.concurrency repo review

2014-08-11 Thread Joshua Harlow



On Mon, Aug 11, 2014 at 11:02 AM, Yuriy Taraday yorik@gmail.com 
wrote:
On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow harlo...@outlook.com 
wrote:

One question from me:

Will there be later fixes to remove oslo.config dependency/usage 
from oslo.concurrency?


I still don't understand how oslo.concurrency can be used as a 
library with the configuration being set in a static manner via 
oslo.config (let's use the example of `lock_path` @ 
https://github.com/YorikSar/oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
For example:


Library X inside application Z uses lockutils (via the nice 
oslo.concurrency library) and sets the configuration `lock_path` to 
its desired settings, then library Y (also a user of 
oslo.concurrency) inside same application Z sets the configuration 
for `lock_path` to its desired settings. Now both have some unknown 
set of configuration they have set and when library X (or Y) 
continues to use lockutils they will be using some mix of 
configuration (likely some mish mash of settings set by X and Y); 
perhaps to a `lock_path` that neither actually wants to be able to 
write to...


This doesn't seem like it will end well; and will just cause 
headaches during debug sessions, testing, integration and more...


The same question can be asked about the `set_defaults()` function, 
how is library Y or X expected to use this (are they?)??


I hope one of the later changes is to remove/fix this??

Thoughts?

-Josh


I'd be happy to remove lock_path config variable altogether. It's 
basically never used. There are two basic branches in code wrt 
lock_path:
- when you provide lock_path argument to lock (and derivative 
functions), file-based lock is used and CONF.lock_path is ignored; 
- when you don't provide lock_path in arguments, semaphore-based lock 
is used and CONF.lock_path is just a prefix for its name (before 
hashing).


Agreed, it just seems confusing (and bad) to have parts of the API come 
in from `CONF.lock_path` (or other `CONF.*` options) and other parts of 
the API come in via function parameters. This just makes understanding 
the API and knowing how to interact with it that much harder (after all 
what is the right way of using XYZ feature when it can be changed via a 
out-of-band *hidden* API call via configuration adjustments under the 
covers?)... This makes it really hard to use oslo.concurrency in 
taskflow (and likely other libraries that would like to consume 
oslo.concurrency, seeing that it will be on pypi, I would expect this 
number to grow...) since taskflow would really appreciate and desire to 
have stable APIs that don't change by some configuration that can be 
set by some party via some out-of-band method (for example some other 
library or program calling `set_defaults()`). This kind of way of using 
an API (half of the settings from config, half of the settings from the 
functions API...) may be ok for applications but it's not IMHO ok for 
libraries (or clients) that want to use oslo.concurrency. 

Hopefully it can be fixed some that it works via both ways? Oslo.db I 
believe made this work better by allowing for configuration to come in 
via a configuration object that can be provided by the user of oslo.db, 
this makes the API that oslo.db exposes strongly tied to the attributes 
 documentation of that object. I still don't think thats perfect 
either since its likely that the documentation for what that objects 
attributes should be is not as update to date or easy to change as 
updating function/method documentation...


I wonder if users even set lock_path in their configs as it has 
almost no effect. So I'm all for removing it, but...
From what I understand, every major change in lockutils drags along a 
lot of headache for everybody (and risk of bugs that would be 
discovered very late). So is such change really worth it? And if so, 
it will require very thorough research of lockutils usage patterns.


Sounds like tech debt to me, it always requires work to make something 
better. Are we the type of community that will avoid changing things 
(for the better) because we fear introducing new bugs that may be found 
along the way? I for one hope that we are not that type of community 
(that type of community will die due to its own *fake* fears...).




--

Kind regards, Yuriy.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] oslo.concurrency repo review

2014-08-11 Thread Doug Hellmann

On Aug 11, 2014, at 3:26 PM, Joshua Harlow harlo...@outlook.com wrote:

 
 
 On Mon, Aug 11, 2014 at 11:02 AM, Yuriy Taraday yorik@gmail.com wrote:
 On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow harlo...@outlook.com wrote:
 One question from me:
 Will there be later fixes to remove oslo.config dependency/usage from 
 oslo.concurrency?
 I still don't understand how oslo.concurrency can be used as a library with 
 the configuration being set in a static manner via oslo.config (let's use 
 the example of `lock_path` @ 
 https://github.com/YorikSar/oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41).
  For example:
 Library X inside application Z uses lockutils (via the nice 
 oslo.concurrency library) and sets the configuration `lock_path` to its 
 desired settings, then library Y (also a user of oslo.concurrency) inside 
 same application Z sets the configuration for `lock_path` to its desired 
 settings. Now both have some unknown set of configuration they have set and 
 when library X (or Y) continues to use lockutils they will be using some 
 mix of configuration (likely some mish mash of settings set by X and Y); 
 perhaps to a `lock_path` that neither actually wants to be able to write 
 to...
 This doesn't seem like it will end well; and will just cause headaches 
 during debug sessions, testing, integration and more...
 The same question can be asked about the `set_defaults()` function, how is 
 library Y or X expected to use this (are they?)??
 I hope one of the later changes is to remove/fix this??
 Thoughts?
 -Josh
 I'd be happy to remove lock_path config variable altogether. It's basically 
 never used. There are two basic branches in code wrt lock_path:
 - when you provide lock_path argument to lock (and derivative functions), 
 file-based lock is used and CONF.lock_path is ignored; - when you don't 
 provide lock_path in arguments, semaphore-based lock is used and 
 CONF.lock_path is just a prefix for its name (before hashing).
 
 Agreed, it just seems confusing (and bad) to have parts of the API come in 
 from `CONF.lock_path` (or other `CONF.*` options) and other parts of the API 
 come in via function parameters. This just makes understanding the API and 
 knowing how to interact with it that much harder (after all what is the right 
 way of using XYZ feature when it can be changed via a out-of-band *hidden* 
 API call via configuration adjustments under the covers?)... This makes it 
 really hard to use oslo.concurrency in taskflow (and likely other libraries 
 that would like to consume oslo.concurrency, seeing that it will be on pypi, 
 I would 

The libraries placed in the oslo namespace are very much NOT meant to be used 
by anything other than OpenStack. They are intended to be the glue layer 
between OpenStack and some other implementation libraries.

oslo.concurrency wraps pylockfile and the plan is to move the actual lock code 
into pylockfile without the oslo.config dependency. That will make pylockfile 
reusable by taskflow and tooz, and the locking stuff in oslo.concurrency a 
smaller API with consistent configuration for use by applications.

 expect this number to grow...) since taskflow would really appreciate and 
 desire to have stable APIs that don't change by some configuration that can 
 be set by some party via some out-of-band method (for example some other 
 library or program calling `set_defaults()`). This kind of way of using an 
 API (half of the settings from config, half of the settings from the 
 functions API...) may be ok for applications but it's not IMHO ok for 
 libraries (or clients) that want to use oslo.concurrency. 
 Hopefully it can be fixed some that it works via both ways? Oslo.db I believe 
 made this work better by allowing for configuration to come in via a 
 configuration object that can be provided by the user of oslo.db, this makes 
 the API that oslo.db exposes strongly tied to the attributes  documentation 
 of that object. I still don't think thats perfect either since its likely 
 that the documentation for what that objects attributes should be is not as 
 update to date or easy to change as updating function/method documentation…

That technique of having the configuration object passed to the oslo library 
will be repeated in the other new libraries we are creating if they already 
depend on configuration settings of some sort. The configuration options are 
not part of the public API of the library, so they and their definitions will 
be hidden from the caller, but the library has to be given a configuration 
object in order to load the settings for itself.

 
 I wonder if users even set lock_path in their configs as it has almost no 
 effect. So I'm all for removing it, but...
 From what I understand, every major change in lockutils drags along a lot of 
 headache for everybody (and risk of bugs that would be discovered very 
 late). So is such change really worth it? And if so, it will require very 
 thorough 

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Sandy Walsh
On 8/11/2014 4:22 PM, Eoghan Glynn wrote:

 Hi Eoghan,

 Thanks for the note below. However, one thing the overview below does not
 cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged. Many
 folks feel that this technology is a viable solution for the problem space
 discussed below.
 Great question Brad!

 As it happens we've been working closely with Paul Dix (lead
 developer of InfluxDB) to ensure that this metrics store would be
 usable as a backend driver. That conversation actually kicked off
 at the Juno summit in Atlanta, but it really got off the ground
 at our mid-cycle meet-up in Paris on in early July.
...

 The InfluxDB folks have committed to implementing those features in
 over July and August, and have made concrete progress on that score.

 I hope that provides enough detail to answer to your question?

I guess it begs the question, if influxdb will do what you want and it's
open source (MIT) as well as commercially supported, how does gnocchi
differentiate?

 Cheers,
 Eoghan

 Thanks,

 Brad


 Brad Topol, Ph.D.
 IBM Distinguished Engineer
 OpenStack
 (919) 543-0646
 Internet: bto...@us.ibm.com
 Assistant: Kendra Witherspoon (919) 254-0680



 From: Eoghan Glynn egl...@redhat.com
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org,
 Date: 08/06/2014 11:17 AM
 Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
 project





 Folks,

 It's come to our attention that some key individuals are not
 fully up-to-date on gnocchi activities, so it being a good and
 healthy thing to ensure we're as communicative as possible about
 our roadmap, I've provided a high-level overview here of our
 thinking. This is intended as a precursor to further discussion
 with the TC.

 Cheers,
 Eoghan


 What gnocchi is:
 ===

 Gnocchi is a separate, but related, project spun up on stackforge
 by Julien Danjou, with the objective of providing efficient
 storage and retrieval of timeseries-oriented data and resource
 representations.

 The goal is to experiment with a potential approach to addressing
 an architectural misstep made in the very earliest days of
 ceilometer, specifically the decision to store snapshots of some
 resource metadata alongside each metric datapoint. The core idea
 is to move to storing datapoints shorn of metadata, and instead
 allow the resource-state timeline to be reconstructed more cheaply
 from much less frequently occurring events (e.g. instance resizes
 or migrations).


 What gnocchi isn't:
 ==

 Gnocchi is not a large-scale under-the-radar rewrite of a core
 OpenStack component along the lines of keystone-lite.

 The change is concentrated on the final data-storage phase of
 the ceilometer pipeline, so will have little initial impact on the
 data-acquiring agents, or on transformation phase.

 We've been totally open at the Atlanta summit and other forums
 about this approach being a multi-cycle effort.


 Why we decided to do it this way:
 

 The intent behind spinning up a separate project on stackforge
 was to allow the work progress at arms-length from ceilometer,
 allowing normalcy to be maintained on the core project and a
 rapid rate of innovation on gnocchi.

 Note that that the developers primarily contributing to gnocchi
 represent a cross-section of the core team, and there's a regular
 feedback loop in the form of a recurring agenda item at the
 weekly team meeting to avoid the effort becoming silo'd.


 But isn't re-architecting frowned upon?
 ==

 Well, the architecture of other OpenStack projects have also
 under-gone change as the community understanding of the
 implications of prior design decisions has evolved.

 Take for example the move towards nova no-db-compute  the
 unified-object-model in order to address issues in the nova
 architecture that made progress towards rolling upgrades
 unneccessarily difficult.

 The point, in my understanding, is not to avoid doing the
 course-correction where it's deemed necessary. Rather, the
 principle is more that these corrections happen in an open
 and planned way.


 The path forward:
 

 A subset of the ceilometer community will continue to work on
 gnocchi in parallel with the ceilometer core over the remainder
 of the Juno cycle and into the Kilo timeframe. The goal is to
 have an initial implementation of gnocchi ready for tech preview
 by the end of Juno, and to have the integration/migration/
 co-existence questions addressed in Kilo.

 Moving the ceilometer core to using gnocchi will be contingent
 on it demonstrating the required performance characteristics and
 providing the semantics needed to support a v3 ceilometer API
 that's fit-for-purpose.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 

Re: [openstack-dev] [oslo] oslo.concurrency repo review

2014-08-11 Thread Ben Nemec
On 08/11/2014 02:20 PM, Joshua Harlow wrote:
 
 
 On Mon, Aug 11, 2014 at 11:39 AM, Ben Nemec openst...@nemebean.com 
 wrote:
 On 08/11/2014 01:02 PM, Yuriy Taraday wrote:
  On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow 
 harlo...@outlook.com wrote:
  
  One question from me:

  Will there be later fixes to remove oslo.config dependency/usage 
 from
  oslo.concurrency?

  I still don't understand how oslo.concurrency can be used as a 
 library
  with the configuration being set in a static manner via 
 oslo.config (let's
  use the example of `lock_path` @ https://github.com/YorikSar/
  oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
 For
  example:

  Library X inside application Z uses lockutils (via the nice
  oslo.concurrency library) and sets the configuration `lock_path` 
 to its
  desired settings, then library Y (also a user of oslo.concurrency) 
 inside
  same application Z sets the configuration for `lock_path` to its 
 desired
  settings. Now both have some unknown set of configuration they 
 have set and
  when library X (or Y) continues to use lockutils they will be 
 using some
  mix of configuration (likely some mish mash of settings set by X 
 and Y);
  perhaps to a `lock_path` that neither actually wants to be able to 
 write
  to...

  This doesn't seem like it will end well; and will just cause 
 headaches
  during debug sessions, testing, integration and more...

  The same question can be asked about the `set_defaults()` 
 function, how is
  library Y or X expected to use this (are they?)??

  I hope one of the later changes is to remove/fix this??

  Thoughts?

  -Josh
  
  
  I'd be happy to remove lock_path config variable altogether. It's 
 basically
  never used. There are two basic branches in code wrt lock_path:
  - when you provide lock_path argument to lock (and derivative 
 functions),
  file-based lock is used and CONF.lock_path is ignored;
  - when you don't provide lock_path in arguments, semaphore-based 
 lock is
  used and CONF.lock_path is just a prefix for its name (before 
 hashing).
  
  I wonder if users even set lock_path in their configs as it has 
 almost no
  effect. So I'm all for removing it, but...
  From what I understand, every major change in lockutils drags along 
 a lot
  of headache for everybody (and risk of bugs that would be 
 discovered very
  late). So is such change really worth it? And if so, it will 
 require very
  thorough research of lockutils usage patterns.

 Two things lock_path has to stay for: Windows and consumers who 
 require
 file-based locking semantics.  Neither of those use cases are trivial 
 to
 remove, so IMHO it would not be appropriate to do it as part of the
 graduation.  If we were going to alter the API that much it needed to
 happen in incubator.


 As far as lock_path mismatches, that shouldn't be a problem unless a
 consumer is doing something very unwise.  Oslo libs get their
 configuration from the application using them, so unless the 
 application
 passes two separate conf objects to library X and Y they're both going
 to get consistent settings.  If someone _is_ doing that, then I think
 it's their responsibility to make sure the options in both config 
 files
 are compatible with each other.
 
 Why would it be assumed they would pass the same settings (how is that 
 even possible to know ahead of time? especially if library X pulls in a 
 new library ZZ that requires a new configuration setting). For example, 
 one directory for `lock_path` may be reasonable for tooz and another 
 may be reasonable for taskflow (completly depends on there intended 
 usage), it would not likely desireable to have them go to the same 
 location. 

The only reason I can see that you would want to do this is to avoid
lock name collisions, but I think I'd rather namespace lib locks than
require users to manage multiple lock paths.  Even the one lock path has
been a hassle.

 Forcing application Z to know the inner workings of library X 
 and library Y (or future unknown library ZZ) is just pushing the 
 problem onto the library user, which seems inappropriate and breaks the 
 whole point of having abstractions  APIs in the first place... This 
 IMHO is part of the problem with having statically set *action at a 
 distance* type of configuration, the libraries themselves are not in 
 control of their own configuration, which breaks abstractions  APIs 
 left and right. If some application Y can go under a library and pull 
 the rug out from under it, how is that a reasonable thing to expect the 
 library to be able to predict  handle?

The application doesn't have to, and shouldn't, know about lib options.
 The libraries have a list_opts method that returns the list of options
the lib needs, which is sufficient for the config generator to do the
right thing (a bit of an oversimplification maybe, but the point is
there's an API for handling the options so the app doesn't need to).
Besides that the application doesn't 

Re: [openstack-dev] [oslo] oslo.concurrency repo review

2014-08-11 Thread Joshua Harlow



On Mon, Aug 11, 2014 at 12:47 PM, Doug Hellmann d...@doughellmann.com 
wrote:


On Aug 11, 2014, at 3:26 PM, Joshua Harlow harlo...@outlook.com 
wrote:


 
 
 On Mon, Aug 11, 2014 at 11:02 AM, Yuriy Taraday 
yorik@gmail.com wrote:
 On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow 
harlo...@outlook.com wrote:

 One question from me:
 Will there be later fixes to remove oslo.config dependency/usage 
from oslo.concurrency?
 I still don't understand how oslo.concurrency can be used as a 
library with the configuration being set in a static manner via 
oslo.config (let's use the example of `lock_path` @ 
https://github.com/YorikSar/oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
For example:
 Library X inside application Z uses lockutils (via the nice 
oslo.concurrency library) and sets the configuration `lock_path` 
to its desired settings, then library Y (also a user of 
oslo.concurrency) inside same application Z sets the configuration 
for `lock_path` to its desired settings. Now both have some 
unknown set of configuration they have set and when library X (or 
Y) continues to use lockutils they will be using some mix of 
configuration (likely some mish mash of settings set by X and Y); 
perhaps to a `lock_path` that neither actually wants to be able to 
write to...
 This doesn't seem like it will end well; and will just cause 
headaches during debug sessions, testing, integration and more...
 The same question can be asked about the `set_defaults()` 
function, how is library Y or X expected to use this (are they?)??

 I hope one of the later changes is to remove/fix this??
 Thoughts?
 -Josh
 I'd be happy to remove lock_path config variable altogether. It's 
basically never used. There are two basic branches in code wrt 
lock_path:
 - when you provide lock_path argument to lock (and derivative 
functions), file-based lock is used and CONF.lock_path is ignored; 
- when you don't provide lock_path in arguments, semaphore-based 
lock is used and CONF.lock_path is just a prefix for its name 
(before hashing).
 
 Agreed, it just seems confusing (and bad) to have parts of the API 
come in from `CONF.lock_path` (or other `CONF.*` options) and other 
parts of the API come in via function parameters. This just makes 
understanding the API and knowing how to interact with it that much 
harder (after all what is the right way of using XYZ feature when it 
can be changed via a out-of-band *hidden* API call via configuration 
adjustments under the covers?)... This makes it really hard to use 
oslo.concurrency in taskflow (and likely other libraries that would 
like to consume oslo.concurrency, seeing that it will be on pypi, I 
would 


The libraries placed in the oslo namespace are very much NOT meant to 
be used by anything other than OpenStack. They are intended to be the 
glue layer between OpenStack and some other implementation libraries.


oslo.concurrency wraps pylockfile and the plan is to move the actual 
lock code into pylockfile without the oslo.config dependency. That 
will make pylockfile reusable by taskflow and tooz, and the locking 
stuff in oslo.concurrency a smaller API with consistent configuration 
for use by applications.


Sounds great, I've been wondering why 
https://github.com/stackforge/tooz/commit/f3e11e40f9871f8328 
happened/merged (maybe it should be changed?). I see that 
https://review.openstack.org/#/c/102202/ merged so that's good news and 
hopefully makes the underlying lockutils functionality more useful to 
outside of openstack users in the near-term future (which includes 
taskflow, being that it is being used in  outside openstack by various 
entities).





 expect this number to grow...) since taskflow would really 
appreciate and desire to have stable APIs that don't change by some 
configuration that can be set by some party via some out-of-band 
method (for example some other library or program calling 
`set_defaults()`). This kind of way of using an API (half of the 
settings from config, half of the settings from the functions 
API...) may be ok for applications but it's not IMHO ok for 
libraries (or clients) that want to use oslo.concurrency. 
 Hopefully it can be fixed some that it works via both ways? Oslo.db 
I believe made this work better by allowing for configuration to 
come in via a configuration object that can be provided by the user 
of oslo.db, this makes the API that oslo.db exposes strongly tied to 
the attributes  documentation of that object. I still don't think 
thats perfect either since its likely that the documentation for 
what that objects attributes should be is not as update to date or 
easy to change as updating function/method documentation…


That technique of having the configuration object passed to the oslo 
library will be repeated in the other new libraries we are creating 
if they already depend on configuration settings of some sort. The 
configuration options are not part of the public API of the library, 
so they 

Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Eugene Nikanorov
Hi folks,

That actually going in opposite direction to what flavor framework is
trying to do (and for dispatching it's doing the same as providers). REST
call dispatching should really go via the root object.

I don't quite get the issue with health monitors. If HM is incorrectly
configured prior to association with a pool - API layer should handle that.
I don't think driver implementations should be different at constraints to
HM parameters.

So I'm -1 on adding provider (or flavor) to each entity. After all, it
looks just like data denormalization which actually will affect lots of API
aspects in negative way.

Thanks,
Eugene.



On Mon, Aug 11, 2014 at 11:20 PM, Vijay Venkatachalam 
vijay.venkatacha...@citrix.com wrote:


 Yes, the point was to say the plugin need not restrict and let driver
 decide what to do with the API.

 Even if the call was made to driver instantaneously, I understand, the
 driver might decide to ignore
 first and schedule later. But, if the call is present, there is scope for
 validation.
 Also, the driver might be scheduling an async-api to backend, in which
 case  deployment error
 cannot be shown to the user instantaneously.

 W.r.t. identifying a provider/driver, how would it be to make tenant the
 default root object?
 tenantid is already associated with each of these entities, so no
 additional pain.
 For the tenant who wants to override let him specify provider in each of
 the entities.
 If you think of this in terms of the UI, let's say if the loadbalancer
 configuration is exposed
 as a single wizard (which has loadbalancer, listener, pool, monitor
 properties) then provider
  is chosen only once.

 Curious question, is flavour framework expected to address this problem?

 Thanks,
 Vijay V.

 -Original Message-
 From: Doug Wiegley [mailto:do...@a10networks.com]
 Sent: 11 August 2014 22:02
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling
 driver interface on every API request

 Hi Sam,

 Very true.  I think that Vijay’s objection is that we are currently
 imposing a logical structure on the driver, when it should be a driver
 decision.  Certainly, it goes both ways.

 And I also agree that the mechanism for returning multiple errors, and the
 ability to specify whether those errors are fatal or not, individually, is
 currently weak.

 Doug


 On 8/11/14, 10:21 AM, Samuel Bercovici samu...@radware.com wrote:

 Hi Doug,
 
 In some implementations Driver !== Device. I think this is also true
 for HA Proxy.
 This might mean that there is a difference between creating a logical
 object and when there is enough information to actually schedule/place
 this into a device.
 The ability to express such errors (detecting an error on a logical
 object after it was created but when it actually get scheduled) should
 be discussed and addressed anyway.
 
 -Sam.
 
 
 -Original Message-
 From: Doug Wiegley [mailto:do...@a10networks.com]
 Sent: Monday, August 11, 2014 6:55 PM
 To: OpenStack Development Mailing List (not for usage questions)
 Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling
 driver interface on every API request
 
 Hi all,
 
  Validations such as ³timeout  delay² should be performed on the API
 level before it reaches the driver.
 For a configuration tree (lb, listeners, pools, etc.), there should be
 one provider.
 
 You¹re right, but I think the point of Vijay¹s example was to highlight
 the combo error problem with populating all of the driver objects at
 once (in short, the driver interface isn¹t well suited to that model.)
 That his one example can be covered by API validators is irrelevant.
 Consider a backend that does not support APP_COOKIE¹s, or
 HTTPS_TERMINATED (but has multiple listeners) instead.  Should the
 entire load balancer create fail, or should it offer degraded service?
 Do all drivers have to implement a transaction rollback; wait, the
 interface makes that very hard.  That¹s his point.  The driver is no
 longer just glue code between interfaces; it¹s now a mini-object error
 handler.
 
 
  Having provider defined in multiple places does not make sense.
 
 Channeling Brandon, who can yell if I get this wrong, the point is not
 to have a potentially different provider on each object.  It¹s to allow
 a provider to be assigned when the first object in the tree is created,
 so that future related objects will always get routed to the same
 provider.
 Not knowing which provider should get all the objects is why we have to
 wait until we see a LoadBalancer object.
 
 
 All of this sort of edge case nonsense is because we (the royal we, the
 community), wanted all load balancer objects to be ³root² objects, even
 though only one of them is an actual root today, to support
 many-to-many relationships among all of them, at some future date,
 without an interface change.  If my bias is showing that I¹m not a fan
 of adding this complexity 

Re: [openstack-dev] Which program for Rally

2014-08-11 Thread Matthew Treinish
I apologize for the delay in my response to this thread, between travelling
and having a stuck 'a' key on my laptop this is the earliest I could respond.
I opted for a separate branch on this thread to summarize my views and I'll
respond inline later on some of the previous discussion.

On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
 Hi everyone,
 
 At the TC meeting yesterday we discussed Rally program request and
 incubation request. We quickly dismissed the incubation request, as
 Rally appears to be able to live happily on top of OpenStack and would
 benefit from having a release cycle decoupled from the OpenStack
 integrated release.
 
 That leaves the question of the program. OpenStack programs are created
 by the Technical Committee, to bless existing efforts and teams that are
 considered *essential* to the production of the OpenStack integrated
 release and the completion of the OpenStack project mission. There are 3
 ways to look at Rally and official programs at this point:
 
 1. Rally as an essential QA tool
 Performance testing (and especially performance regression testing) is
 an essential QA function, and a feature that Rally provides. If the QA
 team is happy to use Rally to fill that function, then Rally can
 obviously be adopted by the (already-existing) QA program. That said,
 that would put Rally under the authority of the QA PTL, and that raises
 a few questions due to the current architecture of Rally, which is more
 product-oriented. There needs to be further discussion between the QA
 core team and the Rally team to see how that could work and if that
 option would be acceptable for both sides.

So ideally this is where Rally would belong, the scope of what Rally is
attempting to do is definitely inside the scope of the QA program. I don't see
any reason why that isn't the case. The issue is with what Rally is in it's
current form. It's scope is too large and monolithic, and it duplicates much of
the functionality we either already have or need in current QA or Infra
projects. But, nothing in Rally is designed to be used outside of it. I actually
feel pretty strongly that in it's current form Rally should *not* be a part of
any OpenStack program.

All of the points Sean was making in the other branch on this thread (which I'll
probably respond to later) are a huge concerns I share with Rally. He basically
summarized most of my views on the topic, so I'll try not to rewrite everything.
But, the fact that all of this duplicate functionality was implemented in a
completely separate manner which is Rally specific and can't really be used
unless all of Rally is used is of a large concern. What I think the path
forward here is to have both QA and Rally work together on getting common
functionality that is re-usable and shareable. Additionally, I have some
concerns over the methodology that Rally uses for it's performance measurement.
But, I'll table that discussion because I think it would partially derail this
discussion.

So one open question is long-term where would this leave Rally if we want to
bring it in under the QA program. (after splitting up the functionality to more
conducive with all our existing tools and projects) The one thing Rally does
here which we don't have an analogous solution for is, for lack of better term,
the post processing layer. The part that generates the performs the analysis on
the collected data and generates the graphs. That is something that we'll have
an eventually need for and that is something that we can work on turning rally
into as we migrate everything to actually work together.

There are probably also other parts of Rally which don't fit into an existing
QA program project, (or the QA program in general) and in those cases we
probably should split them off as smaller projects to implement that bit. For
example, the SLA stuff Rally has that probably should be a separate entity as
well, but I'm unsure if that fits under QA program.

My primary concern is the timing for doing all of this work. We're approaching
J-3 and honestly this feels like it would take the better part of an entire
cycle to analyze, plan, and then implement. Starting an analysis of how to do
all of the work at this point I feel would just distract everyone from
completing our dev goals for the cycle. Probably the Rally team, if they want
to move forward here, should start the analysis of this structural split and we
can all pick this up together post-juno.

 
 2. Rally as an essential operator tool
 Regular benchmarking of OpenStack deployments is a best practice for
 cloud operators, and a feature that Rally provides. With a bit of a
 stretch, we could consider that benchmarking is essential to the
 completion of the OpenStack project mission. That program could one day
 evolve to include more such operations best practices tools. In
 addition to the slight stretch already mentioned, one concern here is
 that we still want to have performance testing in QA 

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Eoghan Glynn


 On 8/11/2014 4:22 PM, Eoghan Glynn wrote:
 
  Hi Eoghan,
 
  Thanks for the note below. However, one thing the overview below does not
  cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged.
  Many
  folks feel that this technology is a viable solution for the problem space
  discussed below.
  Great question Brad!
 
  As it happens we've been working closely with Paul Dix (lead
  developer of InfluxDB) to ensure that this metrics store would be
  usable as a backend driver. That conversation actually kicked off
  at the Juno summit in Atlanta, but it really got off the ground
  at our mid-cycle meet-up in Paris on in early July.
 ...
 
  The InfluxDB folks have committed to implementing those features in
  over July and August, and have made concrete progress on that score.
 
  I hope that provides enough detail to answer to your question?
 
 I guess it begs the question, if influxdb will do what you want and it's
 open source (MIT) as well as commercially supported, how does gnocchi
 differentiate?

Hi Sandy,

One of the ideas behind gnocchi is to combine resource representation
and timeseries-oriented storage of metric data, providing an efficient
and convenient way to query for metric data associated with individual
resources.

Also, having an API layered above the storage driver avoids locking in
directly with a particular metrics-oriented DB, allowing for the
potential to support multiple storage driver options (e.g. to choose
between a canonical implementation based on Swift, an InfluxDB driver,
and an OpenTSDB driver, say).

A less compelling reason would be to provide a well-defined hook point
to innovate with aggregation/analytic logic not supported natively
in the underlying drivers (e.g. period-spanning statistics such as
exponentially-weighted moving average or even Holt-Winters).

Cheers,
Eoghan

 
  Cheers,
  Eoghan
 
  Thanks,
 
  Brad
 
 
  Brad Topol, Ph.D.
  IBM Distinguished Engineer
  OpenStack
  (919) 543-0646
  Internet: bto...@us.ibm.com
  Assistant: Kendra Witherspoon (919) 254-0680
 
 
 
  From: Eoghan Glynn egl...@redhat.com
  To: OpenStack Development Mailing List (not for usage questions)
  openstack-dev@lists.openstack.org,
  Date: 08/06/2014 11:17 AM
  Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
  project
 
 
 
 
 
  Folks,
 
  It's come to our attention that some key individuals are not
  fully up-to-date on gnocchi activities, so it being a good and
  healthy thing to ensure we're as communicative as possible about
  our roadmap, I've provided a high-level overview here of our
  thinking. This is intended as a precursor to further discussion
  with the TC.
 
  Cheers,
  Eoghan
 
 
  What gnocchi is:
  ===
 
  Gnocchi is a separate, but related, project spun up on stackforge
  by Julien Danjou, with the objective of providing efficient
  storage and retrieval of timeseries-oriented data and resource
  representations.
 
  The goal is to experiment with a potential approach to addressing
  an architectural misstep made in the very earliest days of
  ceilometer, specifically the decision to store snapshots of some
  resource metadata alongside each metric datapoint. The core idea
  is to move to storing datapoints shorn of metadata, and instead
  allow the resource-state timeline to be reconstructed more cheaply
  from much less frequently occurring events (e.g. instance resizes
  or migrations).
 
 
  What gnocchi isn't:
  ==
 
  Gnocchi is not a large-scale under-the-radar rewrite of a core
  OpenStack component along the lines of keystone-lite.
 
  The change is concentrated on the final data-storage phase of
  the ceilometer pipeline, so will have little initial impact on the
  data-acquiring agents, or on transformation phase.
 
  We've been totally open at the Atlanta summit and other forums
  about this approach being a multi-cycle effort.
 
 
  Why we decided to do it this way:
  
 
  The intent behind spinning up a separate project on stackforge
  was to allow the work progress at arms-length from ceilometer,
  allowing normalcy to be maintained on the core project and a
  rapid rate of innovation on gnocchi.
 
  Note that that the developers primarily contributing to gnocchi
  represent a cross-section of the core team, and there's a regular
  feedback loop in the form of a recurring agenda item at the
  weekly team meeting to avoid the effort becoming silo'd.
 
 
  But isn't re-architecting frowned upon?
  ==
 
  Well, the architecture of other OpenStack projects have also
  under-gone change as the community understanding of the
  implications of prior design decisions has evolved.
 
  Take for example the move towards nova no-db-compute  the
  unified-object-model in order to address issues in the nova
  architecture that made progress towards rolling upgrades
  unneccessarily difficult.
 
  The point, in my 

Re: [openstack-dev] [Congress] congress-server fails to start

2014-08-11 Thread Peter Balland
Hi Rajdeep,

What version of pip are you running?  Please try installing the latest version 
(https://pip.pypa.io/en/latest/installing.html) and run 'sudo pip install -r 
requirements.txt'.

- Peter

From: Rajdeep Dua rajdeep@gmail.commailto:rajdeep@gmail.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Monday, August 11, 2014 at 11:27 AM
To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Congress] congress-server fails to start

Hi All,
command to start the congress-server fails

$ ./bin/congress-server --config-file etc/congress.conf.sample

Error :
ImportError: No module named keystonemiddleware.auth_token

Installing keystonemiddleware manually also fails

$ sudo pip install keystonemiddleware

Could not find a version that satisfies the requirement oslo.config=1.4.0.0a3 
(from keystonemiddleware) (from versions: )
No distributions matching the version for oslo.config=1.4.0.0a3 (from 
keystonemiddleware)

Thanks
Rajdeep
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

2014-08-11 Thread Zane Bitter

On 11/08/14 14:49, Clint Byrum wrote:

Excerpts from Steven Hardy's message of 2014-08-11 11:40:07 -0700:

On Mon, Aug 11, 2014 at 11:20:50AM -0700, Clint Byrum wrote:

Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:

On 11/08/14 10:46, Clint Byrum wrote:

Right now we're stuck with an update that just doesn't work. It isn't
just about update-failure-recovery, which is coming along nicely, but
it is also about the lack of signals to control rebuild, poor support
for addressing machines as groups, and unacceptable performance in
large stacks.


Are there blueprints/bugs filed for all of these issues?



Convergnce addresses the poor performance for large stacks in general.
We also have this:

https://bugs.launchpad.net/heat/+bug/1306743

Which shows how slow metadata access can get. I have worked on patches
but haven't been able to complete them. We made big strides but we are
at a point where 40 nodes polling Heat every 30s is too much for one CPU


This sounds like the same figure I heard at the design summit; did the 
DB call optimisation work that Steve Baker did immediately after that 
not have any effect?



to handle. When we scaled Heat out onto more CPUs on one box by forking
we ran into eventlet issues. We also ran into issues because even with
many processes we can only use one to resolve templates for a single
stack during update, which was also excessively slow.


Related to this, and a discussion we had recently at the TripleO meetup is
this spec I raised today:

https://review.openstack.org/#/c/113296/

It's following up on the idea that we could potentially address (or at
least mitigate, pending the fully convergence-ified heat) some of these
scalability concerns, if TripleO moves from the one-giant-template model
to a more modular nested-stack/provider model (e.g what Tomas has been
working on)

I've not got into enough detail on that yet to be sure if it's acheivable
for Juno, but it seems initially to be complex-but-doable.

I'd welcome feedback on that idea and how it may fit in with the more
granular convergence-engine model.

Can you link to the eventlet/forking issues bug please?  I thought since
bug #1321303 was fixed that multiple engines and multiple workers should
work OK, and obviously that being true is a precondition to expending
significant effort on the nested stack decoupling plan above.



That was the issue. So we fixed that bug, but we never un-reverted
the patch that forks enough engines to use up all the CPU's on a box
by default. That would likely help a lot with metadata access speed
(we could manually do it in TripleO but we tend to push defaults. :)


Right, and we decided we wouldn't because it's wrong to do that to 
people by default. In some cases the optimal running configuration for 
TripleO will differ from the friendliest out-of-the-box configuration 
for Heat users in general, and in those cases - of which this is one - 
TripleO will need to specify the configuration.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Which program for Rally

2014-08-11 Thread David Kranz

On 08/11/2014 04:21 PM, Matthew Treinish wrote:


I apologize for the delay in my response to this thread, between 
travelling
and having a stuck 'a' key on my laptop this is the earliest I could 
respond.
I opted for a separate branch on this thread to summarize my views and 
I'll

respond inline later on some of the previous discussion.

On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
 Hi everyone,

 At the TC meeting yesterday we discussed Rally program request and
 incubation request. We quickly dismissed the incubation request, as
 Rally appears to be able to live happily on top of OpenStack and would
 benefit from having a release cycle decoupled from the OpenStack
 integrated release.

 That leaves the question of the program. OpenStack programs are created
 by the Technical Committee, to bless existing efforts and teams that are
 considered *essential* to the production of the OpenStack integrated
 release and the completion of the OpenStack project mission. There are 3
 ways to look at Rally and official programs at this point:

 1. Rally as an essential QA tool
 Performance testing (and especially performance regression testing) is
 an essential QA function, and a feature that Rally provides. If the QA
 team is happy to use Rally to fill that function, then Rally can
 obviously be adopted by the (already-existing) QA program. That said,
 that would put Rally under the authority of the QA PTL, and that raises
 a few questions due to the current architecture of Rally, which is more
 product-oriented. There needs to be further discussion between the QA
 core team and the Rally team to see how that could work and if that
 option would be acceptable for both sides.

So ideally this is where Rally would belong, the scope of what Rally is
attempting to do is definitely inside the scope of the QA program. I 
don't see
any reason why that isn't the case. The issue is with what Rally is in 
it's
current form. It's scope is too large and monolithic, and it 
duplicates much of

the functionality we either already have or need in current QA or Infra
projects. But, nothing in Rally is designed to be used outside of it. 
I actually
feel pretty strongly that in it's current form Rally should *not* be a 
part of

any OpenStack program.

All of the points Sean was making in the other branch on this thread 
(which I'll
probably respond to later) are a huge concerns I share with Rally. He 
basically
summarized most of my views on the topic, so I'll try not to rewrite 
everything.
But, the fact that all of this duplicate functionality was implemented 
in a
completely separate manner which is Rally specific and can't really be 
used

unless all of Rally is used is of a large concern. What I think the path
forward here is to have both QA and Rally work together on getting common
functionality that is re-usable and shareable. Additionally, I have some
concerns over the methodology that Rally uses for it's performance 
measurement.
But, I'll table that discussion because I think it would partially 
derail this

discussion.

So one open question is long-term where would this leave Rally if we 
want to
bring it in under the QA program. (after splitting up the 
functionality to more
conducive with all our existing tools and projects) The one thing 
Rally does
here which we don't have an analogous solution for is, for lack of 
better term,
the post processing layer. The part that generates the performs the 
analysis on
the collected data and generates the graphs. That is something that 
we'll have
an eventually need for and that is something that we can work on 
turning rally

into as we migrate everything to actually work together.

There are probably also other parts of Rally which don't fit into an 
existing

QA program project, (or the QA program in general) and in those cases we
probably should split them off as smaller projects to implement that 
bit. For
example, the SLA stuff Rally has that probably should be a separate 
entity as

well, but I'm unsure if that fits under QA program.

My primary concern is the timing for doing all of this work. We're 
approaching
J-3 and honestly this feels like it would take the better part of an 
entire
cycle to analyze, plan, and then implement. Starting an analysis of 
how to do

all of the work at this point I feel would just distract everyone from
completing our dev goals for the cycle. Probably the Rally team, if 
they want
to move forward here, should start the analysis of this structural 
split and we

can all pick this up together post-juno.


 2. Rally as an essential operator tool
 Regular benchmarking of OpenStack deployments is a best practice for
 cloud operators, and a feature that Rally provides. With a bit of a
 stretch, we could consider that benchmarking is essential to the
 completion of the OpenStack project mission. That program could one day
 evolve to include more such operations best practices tools. In
 addition to the slight stretch 

Re: [openstack-dev] [Neutron][LBaaS] Continuing on Calling driver interface on every API request

2014-08-11 Thread Brandon Logan
Hi Eugene,
An example of the HM issue (and really this can happen with any entity)
is if the driver the API sends the configuration to does not actually
support the value of an attribute.

For example: Provider A support PING health monitor type, Provider B
does not.  API allows the PING health monitor type to go through.  Once
a load balancer has been linked with that health monitor and the
LoadBalancer chose to use Provider B, that entire configuration is then
sent to the driver.  The driver errors out not on the LoadBalancer
create, but on the health monitor create.

I think that's the issue.

Thanks,
Brandon

On Tue, 2014-08-12 at 00:17 +0400, Eugene Nikanorov wrote:
 Hi folks,
 
 
 That actually going in opposite direction to what flavor framework is
 trying to do (and for dispatching it's doing the same as providers).
 REST call dispatching should really go via the root object.
 
 
 I don't quite get the issue with health monitors. If HM is incorrectly
 configured prior to association with a pool - API layer should handle
 that.
 I don't think driver implementations should be different at
 constraints to HM parameters.
 
 
 So I'm -1 on adding provider (or flavor) to each entity. After all, it
 looks just like data denormalization which actually will affect lots
 of API aspects in negative way.
 
 
 Thanks,
 Eugene.
 
 
 
 
 On Mon, Aug 11, 2014 at 11:20 PM, Vijay Venkatachalam
 vijay.venkatacha...@citrix.com wrote:
 
 Yes, the point was to say the plugin need not restrict and
 let driver decide what to do with the API.
 
 Even if the call was made to driver instantaneously, I
 understand, the driver might decide to ignore
 first and schedule later. But, if the call is present, there
 is scope for validation.
 Also, the driver might be scheduling an async-api to backend,
 in which case  deployment error
 cannot be shown to the user instantaneously.
 
 W.r.t. identifying a provider/driver, how would it be to make
 tenant the default root object?
 tenantid is already associated with each of these entities,
 so no additional pain.
 For the tenant who wants to override let him specify provider
 in each of the entities.
 If you think of this in terms of the UI, let's say if the
 loadbalancer configuration is exposed
 as a single wizard (which has loadbalancer, listener, pool,
 monitor properties) then provider
  is chosen only once.
 
 Curious question, is flavour framework expected to address
 this problem?
 
 Thanks,
 Vijay V.
 
 -Original Message-
 From: Doug Wiegley [mailto:do...@a10networks.com]
 
 Sent: 11 August 2014 22:02
 To: OpenStack Development Mailing List (not for usage
 questions)
 Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on
 Calling driver interface on every API request
 
 Hi Sam,
 
 Very true.  I think that Vijay’s objection is that we are
 currently imposing a logical structure on the driver, when it
 should be a driver decision.  Certainly, it goes both ways.
 
 And I also agree that the mechanism for returning multiple
 errors, and the ability to specify whether those errors are
 fatal or not, individually, is currently weak.
 
 Doug
 
 
 On 8/11/14, 10:21 AM, Samuel Bercovici samu...@radware.com
 wrote:
 
 Hi Doug,
 
 In some implementations Driver !== Device. I think this is
 also true
 for HA Proxy.
 This might mean that there is a difference between creating a
 logical
 object and when there is enough information to actually
 schedule/place
 this into a device.
 The ability to express such errors (detecting an error on a
 logical
 object after it was created but when it actually get
 scheduled) should
 be discussed and addressed anyway.
 
 -Sam.
 
 
 -Original Message-
 From: Doug Wiegley [mailto:do...@a10networks.com]
 Sent: Monday, August 11, 2014 6:55 PM
 To: OpenStack Development Mailing List (not for usage
 questions)
 Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on
 Calling
 driver interface on every API request
 
 Hi all,
 
  Validations such as ³timeout  delay² should be performed
 on the API
 level before it reaches the driver.
 For a configuration tree (lb, listeners, pools, etc.), there
 should be
 one provider.
 
 You¹re right, but I think the point of Vijay¹s example was to
 highlight
   

Re: [openstack-dev] [TripleO][Nova][Neutron] multiple hypervisors on one compute host - neutron agent and compute hostnames

2014-08-11 Thread Joe Gordon
On Tue, Aug 5, 2014 at 4:17 PM, Robert Collins robe...@robertcollins.net
wrote:

 Hi!

 James has run into an issue implementing the multi-hypervisor spec
 (
 http://git.openstack.org/cgit/openstack/tripleo-specs/tree/specs/juno/tripleo-juno-deploy-cloud-hypervisor-type.rst
 )
 which we're hoping to use to reduce infrastructure overheads by
 deploying OpenStack control plane services in VMs, without requiring
 dedicated VM hypervisor machines in the deploy cloud.

 The issue we've hit is that the Neutron messages for VIF plugging are
 sent to the Neutron agent with an exactly matching hostname to the
 Nova-compute process. However, we have unique hostnames for the
 nova-compute processes on one machine (one for -kvm, one for -docker,
 one for -ironic etc) for a variety of reasons: so we can see if all
 the processes are up, so that we don't get messages for the wrong
 process from nova-api etc.


So you are running multiple nova-computes on a single node? This goes
against the model that nova is operating under. Instead of hacking a
workaround, if we think this is a use case nova/openstack should support,
why not have that discussion before deciding that the best solution is a
shallow patch.




 I think a reasonable step might be to allow the agent host option to
 be a list - e.g.

  [DEFAULT]
  hosts={{nova.compute_hostname}}-libvirt,{{nova.compute_hostname}}-docker

 we'd just make it listen to all the nova-compute hostnames we may have
 on the machine.
 That seems like a fairly shallow patch to me: add a new hosts option
 with no default, change the code to listen to N queues when hosts is
 set, and to report state N times as well (for consistency).
 Alternatively, with a DB migration, we could record N hosts against
 one agent status.

 Alternatively we could run N ovs-agents on one machine (with a
 separate integration bridge each), but I worry that we'd encounter
 unexpected cross-chatter between them on things like external bridge
 flows.

 Thoughts?

 For now, we're going to have to run with a limitation of only one
 vif-plugging hypervisor type per machine - we'll make the agent
 hostname match that of the nova compute that needs VIFs plugged ;)

 -Rob

 --
 Robert Collins rbtcoll...@hp.com
 Distinguished Technologist
 HP Converged Cloud

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Sandy Walsh
On 8/11/2014 5:29 PM, Eoghan Glynn wrote:

 On 8/11/2014 4:22 PM, Eoghan Glynn wrote:
 Hi Eoghan,

 Thanks for the note below. However, one thing the overview below does not
 cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged.
 Many
 folks feel that this technology is a viable solution for the problem space
 discussed below.
 Great question Brad!

 As it happens we've been working closely with Paul Dix (lead
 developer of InfluxDB) to ensure that this metrics store would be
 usable as a backend driver. That conversation actually kicked off
 at the Juno summit in Atlanta, but it really got off the ground
 at our mid-cycle meet-up in Paris on in early July.
 ...
 The InfluxDB folks have committed to implementing those features in
 over July and August, and have made concrete progress on that score.

 I hope that provides enough detail to answer to your question?
 I guess it begs the question, if influxdb will do what you want and it's
 open source (MIT) as well as commercially supported, how does gnocchi
 differentiate?
 Hi Sandy,

 One of the ideas behind gnocchi is to combine resource representation
 and timeseries-oriented storage of metric data, providing an efficient
 and convenient way to query for metric data associated with individual
 resources.

Doesn't InfluxDB do the same?


 Also, having an API layered above the storage driver avoids locking in
 directly with a particular metrics-oriented DB, allowing for the
 potential to support multiple storage driver options (e.g. to choose
 between a canonical implementation based on Swift, an InfluxDB driver,
 and an OpenTSDB driver, say).
Right, I'm not suggesting to remove the storage abstraction layer. I'm
just curious what gnocchi does better/different than InfluxDB?

Or, am I missing the objective here and gnocchi is the abstraction layer
and not an influxdb alternative? If so, my apologies for the confusion.

 A less compelling reason would be to provide a well-defined hook point
 to innovate with aggregation/analytic logic not supported natively
 in the underlying drivers (e.g. period-spanning statistics such as
 exponentially-weighted moving average or even Holt-Winters).
 Cheers,
 Eoghan

  
 Cheers,
 Eoghan

 Thanks,

 Brad


 Brad Topol, Ph.D.
 IBM Distinguished Engineer
 OpenStack
 (919) 543-0646
 Internet: bto...@us.ibm.com
 Assistant: Kendra Witherspoon (919) 254-0680



 From: Eoghan Glynn egl...@redhat.com
 To: OpenStack Development Mailing List (not for usage questions)
 openstack-dev@lists.openstack.org,
 Date: 08/06/2014 11:17 AM
 Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
 project





 Folks,

 It's come to our attention that some key individuals are not
 fully up-to-date on gnocchi activities, so it being a good and
 healthy thing to ensure we're as communicative as possible about
 our roadmap, I've provided a high-level overview here of our
 thinking. This is intended as a precursor to further discussion
 with the TC.

 Cheers,
 Eoghan


 What gnocchi is:
 ===

 Gnocchi is a separate, but related, project spun up on stackforge
 by Julien Danjou, with the objective of providing efficient
 storage and retrieval of timeseries-oriented data and resource
 representations.

 The goal is to experiment with a potential approach to addressing
 an architectural misstep made in the very earliest days of
 ceilometer, specifically the decision to store snapshots of some
 resource metadata alongside each metric datapoint. The core idea
 is to move to storing datapoints shorn of metadata, and instead
 allow the resource-state timeline to be reconstructed more cheaply
 from much less frequently occurring events (e.g. instance resizes
 or migrations).


 What gnocchi isn't:
 ==

 Gnocchi is not a large-scale under-the-radar rewrite of a core
 OpenStack component along the lines of keystone-lite.

 The change is concentrated on the final data-storage phase of
 the ceilometer pipeline, so will have little initial impact on the
 data-acquiring agents, or on transformation phase.

 We've been totally open at the Atlanta summit and other forums
 about this approach being a multi-cycle effort.


 Why we decided to do it this way:
 

 The intent behind spinning up a separate project on stackforge
 was to allow the work progress at arms-length from ceilometer,
 allowing normalcy to be maintained on the core project and a
 rapid rate of innovation on gnocchi.

 Note that that the developers primarily contributing to gnocchi
 represent a cross-section of the core team, and there's a regular
 feedback loop in the form of a recurring agenda item at the
 weekly team meeting to avoid the effort becoming silo'd.


 But isn't re-architecting frowned upon?
 ==

 Well, the architecture of other OpenStack projects have also
 under-gone change as the community understanding of the
 implications of prior design decisions has 

Re: [openstack-dev] [qa][ceilometer] swapping the roles of mongodb and sqlalchemy for ceilometer in Tempest

2014-08-11 Thread Joe Gordon
On Sat, Aug 9, 2014 at 6:29 AM, Eoghan Glynn egl...@redhat.com wrote:



   Hi Folks,
  
   Dina Belova has recently landed some infra patches[1,2] to create
   an experimental mongodb-based Tempest job. This effectively just
   overrides the ceilometer storage backend config so that mongodb
   is used instead of sql-alchemy. The new job has been running
   happily for a few days so I'd like now to consider the path
   forwards with this.
  
   One of our Juno goals under the TC gap analysis was to more fully
   gate against mongodb, given that this is the storage backend
   recommended/supported by many distros. The sql-alchemy backend,
   on the other hand, is more suited for proofs of concept or small
   deployments. However up to now we've been hampered from reflecting
   that reality in the gate, due to the gate being stuck on Precise
   for a long time, as befits LTS, and the version of mongodb needed
   by ceilometer (i.e. 2.4) effectively unavailable on that Ubuntu
   release (in fact it was limited to 2.0.4).
  
   So the orientation towards gating on sql-alchemy was mostly
   driven by legacy issues in the gate's usage of Precise, as
   opposed to this being considered the most logical basket in
   which to put all our testing eggs.
  
   However, we're now finally in the brave new world of Trusty :)
   So I would like to make the long-delayed change over soon.
  
   This would involve transposing the roles of sql-alchemy and
   mongodb in the gate - the mongodb variant becomes the blessed
   job run by default, whereas the sql-alchemy based job to
   relegated to the second tier.
  
   So my questions are:
  
   (a) would the QA side of the house be agreeable to this switch?
  
   and:
  
   (b) how long would the mongodb job need to be stable in this
   experimental mode before we pull the trigger on swicthing?
  
   If the answer to (a) is yes, we can get infra patches proposed
   early next week to make the swap.
  
   Cheers,
   Eoghan
  
   [1]
  
 https://review.openstack.org/#/q/project:openstack-infra/config+branch:master+topic:ceilometer-mongodb-job,n,z
   [2]
  
 https://review.openstack.org/#/q/project:openstack-infra/devstack-gate+branch:master+topic:ceilometer-backend,n,z
  
 
  My interpretation of the gap analysis [1] is merely that you have
 coverage,
  not that you switch to it and relegate the SQLAlchemy tests to second
 chair.
  I believe that's a dangerous departure from current standards. A
 dependency
  on mongodb, due to it's AGPL license, and lack of sufficient support for
 a
  non-AGPL storage back end, has consistently been raised as a blocking
 issue
  for Marconi. [2]

 Sure, the main goal is to have full mongodb-based coverage in the gate.

 So, if the QA/infra folks are prepared to host *both* jobs, then I'd be
 happy to change my request to simply:

   let's promote the mongodb-based Tempest variant to the first tier,
   to run alongside the current sqlalchemy-based job



Ignoring the question of is it ok to say: 'to run ceilometer in any sort of
non-trivial deployment you must manager yet another underlying service,
mongodb' I would prefer not adding an addition gate variant to all
projects.  With the effort to reduce the number of gate variants we have
[0] I would prefer to see just ceilometer gate on both mongodb and
sqlalchemy and the main integrated gate [1] pick just one.

[0] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html
[1]
http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n238



 Does that work for you Devananda?

 Cheers,
 Eoghan

  -Deva
 
 
  [1]
 
 https://wiki.openstack.org/wiki/Governance/TechnicalCommittee/Ceilometer_Gap_Coverage
 
  [2]
 http://lists.openstack.org/pipermail/openstack-dev/2014-March/030510.html
  is a very articulate example of this objection

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


  1   2   >