Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Mark McLoughlin]

On Fri, 2014-08-08 at 09:06 -0400, Russell Bryant wrote:
> On 08/07/2014 08:06 PM, Michael Still wrote:
> > It seems to me that the tension here is that there are groups who
> > would really like to use features in newer libvirts that we don't CI
> > on in the gate. Is it naive to think that a possible solution here is
> > to do the following:
> > 
> >  - revert the libvirt version_cap flag
> 
> I don't feel strongly either way on this.  It seemed useful at the time
> for being able to decouple upgrading libvirt and enabling features that
> come with that.

Right, I suggested the flag as a more deliberate way of avoiding the
issue that was previously seen in the gate with live snapshots. I still
think it's a pretty elegant and useful little feature, and don't think
we need to use it as proxy battle over testing requirements for new
libvirt features.

Mark.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Python3 search path order when building Debian package

[Thomas Goirand]

Hi,

During the build of oslo packages for OpenStack, I'm having the issue
that python3 seems to first look into
/usr/lib/python3.4/dist-packages/olso, finds stuff in it, and then do
not search in the local directory. Because of that, unit tests are
failing during the build. For example, when trying to build
python-oslo.utils (new package, not yet available in Debian), I get:

==
FAIL: unittest.loader.ModuleImportFailure.tests.test_excutils
unittest.loader.ModuleImportFailure.tests.test_excutils
--
_StringException: Traceback (most recent call last):
ImportError: Failed to import test module: tests.test_excutils
Traceback (most recent call last):
  File "/usr/lib/python3.4/unittest/loader.py", line 312, in _find_tests
module = self._get_module_from_name(name)
  File "/usr/lib/python3.4/unittest/loader.py", line 290, in
_get_module_from_name
__import__(name)
  File
"/home/zigo/sources/openstack/juno/python-oslo.utils/build-area/python-oslo.utils-0.1.1/tests/test_excutils.py",
line 22, in 
from oslo.utils import excutils
ImportError: No module named 'oslo.utils'

note that I am running testr the following way:

testr-python3 init
PYTHON=python3.4 testr-python3 run --subunit | subunit2pyunit

Of course, if I first install python3-oslo.utils, then the unit tests
are passing without any issue. Though at build time, the package isn't
installed in the system yet.

What's very surprising to me, is that I have this issue only with Python
3, not with python 2.7, which passes the tests without any issue, and
does the path hunt for the python module correctly.

For other oslo libraries, I have declared some Build-Conflicts, to make
sure that they aren't installed in the system when building. Though that
can't be done with oslo.utils, because it depends on oslo.config and
oslo.i18n.

I think this highlight once more that using namespace should be avoided
if possible. This creates more issues than it solves problems.

Any idea, comment or whatever that may help would be appreciated.

Cheers,

Thomas Goirand (zigo)

P.S: By the way, there's issues with python-oslo.utils with Python 3.4,
after solving the above problem...

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Improvements to current reviews

[Susanne Balle]

I agree with Doug as well. We should update the current patch. Susanne


On Sun, Aug 10, 2014 at 8:18 AM, Gary Kotton  wrote:

>  Hi,
> I took a look at https://review.openstack.org/#/c/105331/ and had one
> minor issue which I think can be addressed. Prior to approving we need to
> understand what the state of the V2 API will be.
> Thanks
> Gary
>
>   From: Vijay Venkatachalam 
> Reply-To: OpenStack List 
> Date: Sunday, August 10, 2014 at 2:57 PM
> To: OpenStack List 
>
> Subject: Re: [openstack-dev] [Neutron][LBaaS] Improvements to current
> reviews
>
>   Thanks Brandon for constant  improvisation.
>
> I agree with Doug. Please update current one. We already hv  more number
> of reviews :-). It will be difficult to manage if we add more.
>
> Thanks,
> Vijay
>
> Sent using CloudMagic
>
> On Sun, Aug 10, 2014 at 3:23 AM, Doug Wiegley 
> wrote:
>
>  I think you should update the current reviews (new patch set, not
> additional review.)
>
> Doug
>
>
> > On Aug 9, 2014, at 3:34 PM, "Brandon Logan" 
> wrote:
> >
> > So I've done some work on improving the code on the current pending
> > reviews.  And would like to get peoples' opinions on whether I should
> > add antoher patch set to those reviews, or add the changes as as another
> > review dependent on the pending ones.
> >
> > To be clear, no matter what the first review in the chain will not
> > change:
> > https://review.openstack.org/#/c/105331/
> >
> > However, if adding another patch set is preferrable then plugin and db
> > implementation review would get another patch set and then obviously
> > anything depending on that.
> >
> > https://review.openstack.org/#/c/105609/
> >
> > My opinion is that I'd like to get both of these in as a new patch set.
> > Reason being that the reviews don't have any +2's and there is
> > uncertainty because of the GBP discussion.  So, I don't think it'd be a
> > major issue if a new patch set was created.  Speak up if you think
> > otherwise.  I'd like to get as many people's thoughts as possible.
> >
> > The changes are:
> >
> > 1) Added data models, which are just plain python object mimicking the
> > sql alchemy models, but don't have the overhead or dynamic nature of
> > being sql alchemy models.  These data models are now returned by the
> > database methods, instead of the sql alchemy objects.  Also, I moved the
> > definition of the sql alchemy models into its own module.  I've been
> > wanting to do this but since I thought I was running out of time I left
> > it for later.
> >
> > These shouldn't cause many merge/rebase conflicts, but it probably cause
> > a few because the sql alchemy models were moved to a different module.
> >
> >
> > 2) The LoadBalancerPluginv2 no longer inherits from the
> > LoadBalancerPluginDbv2.  The database is now a composite attribute of
> > the plugin (i.e. plugin.db.get_loadbalancer()).  This cleans up the code
> > a bit and removes the necessity for _delete_db_entity methods that the
> > drivers needed because now they can actually call the database methods.
> > Also, this makes testing more clear, though I have not added any tests
> > for this because the previous tests are sufficient for now.  Adding the
> > appropriate tests would add 1k or 2k lines most likely.
> >
> > This will likely cause more conflicts because the _db_delete_entity
> > methods have been removed.  However, the new driver interface/mixins
> > defined a db_method for all drivers to use, so if that is being used
> > there shouldn't be any problems.
> >
> > Thanks,
> > Brandon
> >
> >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: FW: [Neutron] Group Based Policy and the way forward

[loy wolfe]

Hi Sumit,

First I want to say I'm not opposed to GBP itself, but has many confusion
about it's core resource model and how it will integrate with neutron core.

Do you mean for whatever GBP backend configured in any future Neutron
deployment, so long as they are in tree, then ML2 core plugin shall always
be there to expose the Neutron core resource: Network/Subnet/Port?

Then since Network/Subnet/Port will never be treated just as LEGACY
COMPATIBLE role, there is no need to extend Nova-Neutron interface to
follow the GBP resource. Anyway, one of optional service plugins inside
Neutron shouldn't has any impact on Nova side.

If we agree on this point, core model of GBP should be reviewed, but not
just focus on naming convention whether it should be called EP or policy
target, then leaving some words here to emphasize GBP is only
complementary. In fact, EP/EPG/BD/RD has been designed to be ABLE TO
REPLACE Neutron core resource, with mapping as the first step to keep
compatible.

In fact, if Neutron core source shall never to be swapped out, GBP core
object could be greatly simplified, because mapping already means
redundancy :-) Only policy-group is meaningful, behind which are very
important policy concept: consumer/producer contracts. After PG is defined,
it should be directly applied to existing Neutron core resource, but not
create similar concept of EP/L2P/L3P, then mapping to them. Mapping is
redundant, and I can understand it's necessity only if someday those
neutron core resource are planned to be swapped out.

Simple conclusion: if GBP is just an optional complementary, then after
defining policy template, directly APPLYING, but not create similar
redundant resource and then MAPPING it to existing neutron core resource.


On Sat, Aug 9, 2014 at 3:35 AM, Jay Pipes  wrote:

> On 08/08/2014 12:29 PM, Sumit Naiksatam wrote:
>
>> Hi Jay, To extend Ivar's response here, the core resources and core
>> plugin configuration does not change with the addition of these
>> extensions. The mechanism to implement the GBP extensions is via a
>> service plugin. So even in a deployment where a GBP service plugin is
>> deployed with a driver which interfaces with a backend that perhaps
>> directly understands some of the GBP constructs, that system would
>> still need to have a core plugin configured that honors Neutron's core
>> resources. Hence my earlier comment that GBP extensions are
>> complementary to the existing core resources (in much the same way as
>> the existing extensions in Neutron).
>>
>
> OK, thanks Sumit. That clearly explains things for me.
>
> Best,
> -jay
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Octavia] Minutes from 8/6/2014 meeting

[Susanne Balle]

Great notes! thanks it helped me catch up after vacation. :)


On Thu, Aug 7, 2014 at 4:33 AM, Stephen Balukoff 
wrote:

> On where to capture notes like this long-term:  I would say the wiki is
> more searchable for now. When we make the transition to IRC meetings, then
> the meeting bots will capture minutes and transcripts in the usual way and
> we can link to these from the wiki.
>
>
> On Thu, Aug 7, 2014 at 1:29 AM, Stephen Balukoff 
> wrote:
>
>> Wow, Trevor! Thanks for capturing all that!
>>
>>
>> On Wed, Aug 6, 2014 at 9:47 PM, Trevor Vardeman <
>> trevor.varde...@rackspace.com> wrote:
>>
>>> Agenda items are numbered, and topics, as discussed, are described
>>> beneath in list format.
>>>
>>> 1) Octavia Constitution and Project Direction Documents (Road map)
>>> a) Constitution and Road map will potentially be adopted after
>>> another couple days; providing those who were busy more time to review the
>>> information
>>>
>>> 2) Octavia Design Proposals
>>> a) Difference between version 0.5 and 1.0 isn't huge
>>> b) Version 2 has many network topology changes and Layer 4 routing
>>> + This includes N node Active-Active
>>> + Would like to avoid Layer 2 connectivity with Load Balancers
>>> (included in version 1 however)
>>> + Layer router driver
>>> + Layer router controller
>>> + Long term solution
>>> c) After refining Version 1 document (with some scrutiny) all
>>> changes will be propagated to the Version 2 document
>>> d) Version 0.5 is unpublished
>>> e) All control layer, anything connected to the intermediate message
>>> bus in version 1, will be collapsed down to 1 daemon.
>>> + No scale-able control, but scale-able service delivery
>>> + Version 1 will be the first large operator compatible version,
>>> that will have both scale-able control and scale-able service delivery
>>> + 0.5 will be a good start
>>> - laying out ground work
>>> - rough topology for the end users
>>> - must be approved by the networking teams for each
>>> contributing company
>>> f) The portions under control of neutron lbaas is the User API and
>>> the driver (for neutron lbaas)
>>> g) If neutron LBaaS is a sufficient front-end (user API doesn't
>>> suck), then Octavia will be kept as a vendor driver
>>> h) Potentially including a REST API on top of Octavia
>>> + Octavia is initially just a vendor driver, no real desire for
>>> another API in front of Octavia
>>> + If someone wants it, the work is "trivial" and can be done in
>>> another project at another time
>>> i) Octavia should have a loose coupling with Neutron; use a shim for
>>> network connectivity (one specifically for Neutron communication in the
>>> start)
>>> + This is going to hold any "dirty hacks" that would be required
>>> to get something done, keeping Octavia clean
>>> - Example: changing the mac address on a port
>>>
>>> 3) Operator Network Topology Requirements
>>> a) One requirement is floating IPs.
>>> b) IPv6 is in demand, but is currently not supported reliably on
>>> Neutron
>>> + IPv6 would be represented as a different load balancer entity,
>>> and possibly include co-location with another Load Balancer
>>> c) Network interface plug-ability (potentially)
>>> d) Sections concerning front-end connectivity should be forwarded to
>>> each company's network specialists for review
>>> + Share findings in the mailing list, and dissect the proposals
>>> with the information and comment what requirements are needing added etc.
>>>
>>> 4) HA/Failover Options/Solutions
>>> a) Rackspace may have a solution to this, but the conversation will
>>> be pushed off to the next meeting (at least)
>>> + Will gather more information from another member in Rackspace
>>> to provide to the ML for initial discussions
>>> b) One option for HA:  Spare pool option (similar to Libra)
>>> + Poor recovery time is a big problem
>>> c) Another option for HA:  Active/Passive
>>> + Bluebox uses one active and one passive configuration, and has
>>> sub-second fail over.  However is not resource-sufficient
>>>
>>> Questions:
>>> Q:  What is the expectation for a release time-frame
>>> A:  Wishful thinking; Octavia version 0.5 beta for Juno (probably not,
>>> but would be awesome to push for that)
>>>
>>> Notes:
>>>  + We need to pressure the Neutron core reviewers to review the Neutron
>>> LBaaS changes to get merges.
>>>  + Version 2 front-end topology is different than the Version 1.  Please
>>> review them individually, and thoroughly
>>>
>>>
>>> PS.  I re-wrote most of the information from the recording (thanks again
>>> Doug).  I have one question for everyone: should I just email this out
>>> after each meeting to the Octavia mailing list, or should I also add it to
>>> a page in an Octavia wiki for Meeting Notes/Minutes or some

[openstack-dev] [Heat] SoftwareDeployment resource is always in progress

[david ferahi]

Hello,

I 'm trying to create a simple stack with heat (Icehouse release).
The template contains SoftwareConfig, SoftwareDeployment and a single
server resources.

The problem is that the SoftwareDeployment resource is always in progress !

After waiting for more than an hour the stack deployment failed and I got
this error:

 TRACE heat.engine.resource HTTPUnauthorized: ERROR: Authentication failed.
Please try again with option --include-password or export
HEAT_INCLUDE_PASSWORD=1
TRACE heat.engine.resource Authentication required

When I checked the log file (/var/log/heat/heat-engine.log), it shows  the
following message(every second):
2014-08-10 19:41:09.622 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10
2014-08-10 19:41:10.648 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10
2014-08-10 19:41:11.671 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10
2014-08-10 19:41:12.690 2391 INFO urllib3.connectionpool [-] Starting new
HTTP connection (1): 192.168.122.10

Here the template I am using :
https://github.com/openstack/heat-templates/blob/master/hot/software-config/example-templates/wordpress/WordPress_software-config_1-instance.yaml

Please help !

Kind Regards,

David
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree

[Thierry Carrez]

gustavo panizzo (gfa) wrote:
> only one think i didn't like it
> 
> why all url,api, etc has to include the word 'preview'?
> i imagine that i would be consuming the new feature using heat, puppet,
> local scripts, custom horizon, whatever. Why do you make me to change
> all them when the feature moves out of preview? it could be a lot of
> rework (for consumers) without gain. I would totally support other big
> fat warnings everywhere (logs, documentation, startup log of
> neutron-server) but don't change the API if is not necessary

I see two issues with this proposal: the first one is what Gustavo just
said: the use of the "preview" package/configoption/API creates friction
when the feature needs to go mainstream (package renaming, change in
configuration for deployers, change in API calls for users...).

My understanding is that the goal is to make it easy for people to "try"
the preview feature, and keeping the experimental feature in-tree is
seen as simpler to experiment with. But the pain from this friction imho
outweighs the pain of deploying an out-of-tree plugin for deployers.

The second issue is that once the feature is in "preview" in tree, it
moves the responsibility/burden of making it official (or removed) to
the core developers (as Salvatore mentioned). I kind of like the
approach where experimental features are developed in faster iterations
out-of-tree and when they are celebrated by experimenters and are ready
to be stable-supported, they are moved in tree.

Regards,

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone] how to generate conf sample in havana?

[Li Tianqing]

Hello,
   Right now, i customerized keystone to meet our requeriments. I want to add 
some configuration options in keystone.conf.sample. But, i 
do not find generate_sample.sh in directory tools in Havana keystone. So i want 
to know how to generate conf sample in havana keystone.
I know, i can write directly to keystone.conf.sample, but should there be 
something like generate_sample.sh in keystone?___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] The future of the integrated release

[Thierry Carrez]

Jeremy Stanley wrote:
> On 2014-08-08 09:43:54 -0700 (-0700), Devananda van der Veen wrote:
> [...]
>> this sounds like it will also solve the current problem when a
>> core reviewer has gone on vacation after blocking something for
>> procedural reasons.
> 
> I don't really think so, unless I'm misunderstanding which vacation
> problem you're talking about. We can either make a vote carry over
> on subsequent patchsets (in which case you need someone to reverse
> or delete the -2 before you can merge it) or not (in which case you
> will lose that -2 on the next uploaded patchset and have to keep
> reapplying it manually).
> 
> Unless perhaps you're suggesting that we should allow approved
> changes to merge even with a workflow -2 from another reviewer
> (keeping in mind that we don't currently allow changes with a
> workflow -1 to merge).

Hmm..

Ideally we'd want a blocking vote that carries over patchsets, but which
can be set or unset by any core/driver/ptl (to be determined). So if the
PTL sets the "procedural" Workflow-2 at feature freeze, I can clear it
when RC1 is published.

Not sure that's supported in Gerrit though :)

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][LBaaS] Use cases with regards to VIP and routers

[Susanne Balle]

Gang,

I was asked the following questions around our Neutron LBaaS use cases:

1.  Will there be a scenario where the “VIP” port will be in a different
Node, from all the Member “VMs” in a pool.

2.  Also how likely is it for the LBaaS configured subnet to not have a
“router” and just use the “extra_routes” option.

3.  Is there a valid use case where customers will be using the
“extra_routes” with subnets instead of the “routers”. ( It would be great
if you have some use case picture for this).

Feel free to chime in here and I'll summaries the answers.

Regards Susanne
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Using host networking for docker containers

[Aleksandr Didenko]

Hi,

we're running only 3 containers in privileged mode: cobbler, rsyslog and
mcollective. Running all the containers in privileged mode is not a good
idea for security reasons. Docker manages DNAT forwarding itself, so it
does not create any overhead for us.

> Is there any real benefits of using separate namespaces in security terms?

Of course, for example only ports specified in EXPOSE line in Dockerfile
are exposed to the host network. So if you start any additional tcp/udp
listeners inside the containers, their ports won't be accessible from the
host network.



On Sat, Aug 9, 2014 at 10:39 AM, Dmitriy Shulyak 
wrote:

> Hi team,
>
> I want to discuss benefits of using host networking [1] for docker
> containers, on master node.
>
> This feature was added in docker 0.11 and basicly means - reuse host
> networking stack, without
> creating separate namespace for each container.
>
> In my opinion it will result in much more stable install/upgrade of master
> node.
>
> 1. There will be no need for dhcrelay/dhcrelay_monitor on host
> 2. No dnat port forwarding
> 3. Performance improvement for pxe boot ???
>
> Is there any real benefits of using separate namespaces in security terms?
>
> To implement this we will need:
>
> 1. Update docker to recent version 0.12/1.x, we will do it anyway, yes?
> 2. Run docker containers with --net=host
>
> Ofcourse it will require running containers in privileged mode, but afaik
> we are already doing this for other reasons.
>
> So, what do you think?
>
> [1] https://github.com/docker/docker/issues/2012
> [2] https://docs.docker.com/articles/networking/
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] The future of the integrated release

[Thierry Carrez]

Eoghan Glynn wrote:
>> The TC has begun to scrutinize new projects more carefully on
>> technical grounds, particularly since some recently-integrated
>> projects have run into scaling limitations that have hampered their
>> adoption. I believe this sort of technical guidance (or curation, if
>> you will) is an essential function of the TC. We've learned that
>> integration bestows legitimacy, as well as assumptions of stability,
>> performance, and scalability, upon projects which are integrated.
>> While that wasn't the intent, I think we need to accept that that is
>> where we stand. We will be faced with some hard decisions regarding
>> projects, both incubated and already integrated, which are clearly not
>> meeting those expectations today.
> 
> How does this relate to the recent gap analysis undertaken by the
> TC for already integrated projects, in order to measure their status
> against the steadily rising bar for integration?

It is almost orthogonal. During the Icehouse cycle the TC created clear
requirements for projects wishing to incubate or graduate for
incubation. But it was unclear how well the *currently-incubated*
projects fulfilled those requirements. Since it was a bit unfair to hold
new entrants to different standards, during the Juno cycle we completed
a gap analysis of currently-integrated projects, created gap coverage
plans to address any identified gap, and reviewed progress on those
after each milestone.

So the idea that being (and remaining) in the integrated release should
also be judged on technical merit is a slightly different effort. It's
always been a factor in our choices, but like Devananda says, it's more
difficult than just checking a number of QA/integration checkboxes. In
some cases, blessing one project in a problem space stifles competition,
innovation and alternate approaches. In some other cases, we reinvent
domain-specific solutions rather than standing on the shoulders of
domain-specific giants in neighboring open source projects.

This all has created a world where you need to be *in* OpenStack to
matter, or to justify the investment. This has created a world where
everything and everyone wants to be in the "OpenStack" integrated
release. This has created more pressure to add new projects, and less
pressure to fix and make the existing projects perfect. 4 years in, we
might want to inflect that trajectory and take steps to fix this world.

-- 
Thierry Carrez (ttx)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova][libvirt] Non-readonly connection to libvirt in unit tests

[Daniel P. Berrange]

On Mon, Aug 04, 2014 at 06:46:13PM -0400, Solly Ross wrote:
> Hi,
> I was wondering if there was a way to get a non-readonly connection
> to libvirt when running the unit tests
> on the CI.  If I call `LibvirtDriver._connect(LibvirtDriver.uri())`,
> it works fine locally, but the upstream
> CI barfs with "libvirtError: internal error Unable to locate libvirtd
> daemon in /usr/sbin (to override, set $LIBVIRTD_PATH to the name of the
> libvirtd binary)".
> If I try to connect by calling libvirt.open(None), it also barfs, saying
> I don't have permission to connect.  I could just set it to always use
> fakelibvirt,
> but it would be nice to be able to run some of the tests against a real
> target.  The tests in question are part of 
> https://review.openstack.org/#/c/111459/,
> and involve manipulating directory-based libvirt storage pools.

Nothing in the unit tests should rely on being able to connect to the
libvirt daemon, as the unit tests should still be able to pass when the
daemon is not running at all. We should be either using fakelibvirt or
mocking any libvirt APIs that need to be tested

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Daniel P. Berrange]

On Mon, Aug 11, 2014 at 08:57:31AM +1000, Michael Still wrote:
> On Sun, Aug 10, 2014 at 2:33 AM, Jeremy Stanley  wrote:
> > On 2014-08-08 09:06:29 -0400 (-0400), Russell Bryant wrote:
> > [...]
> >> We've seen several times that building and maintaining 3rd party
> >> CI is a *lot* of work.
> >
> > Building and maintaining *any* CI is a *lot* of work, not the least
> > of which is the official OpenStack project CI (I believe Monty
> > mentioned in #openstack-infra last night that our CI is about twice
> > the size of Travis-CI now, not sure what metric he's comparing there
> > though).
> >
> >> Like you said in [1], doing this in infra's CI would be ideal. I
> >> think 3rd party should be reserved for when running it in the
> >> project's infrastructure is not an option for some reason
> >> (requires proprietary hw or sw, for example).
> >
> > Add to the "not an option for some reason" list, software which is
> > not easily obtainable through typical installation channels (PyPI,
> > Linux distro-managed package repositories for their LTS/server
> > releases, et cetera) or which requires gyrations which destabilize
> > or significantly complicate maintenance of the overall system as
> > well as reproducibility for developers. It may be possible to work
> > around some of these concerns via access from multiple locations
> > coupled with heavy caching, but adding that in for a one-off source
> > is hard to justify the additional complexity too.
> 
> My understanding is that Fedora has a PPA equivalent which ships a
> "latest and greated" libvirt. So, it would be packages if we went the
> Fedora route, which should be less work.

Yes, there is the 'virt preview' repository

http://fedoraproject.org/wiki/Virtualization_Preview_Repository

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Daniel P. Berrange]

On Fri, Aug 08, 2014 at 09:06:29AM -0400, Russell Bryant wrote:
> On 08/07/2014 08:06 PM, Michael Still wrote:
> > It seems to me that the tension here is that there are groups who
> > would really like to use features in newer libvirts that we don't CI
> > on in the gate. Is it naive to think that a possible solution here is
> > to do the following:
> > 
> >  - revert the libvirt version_cap flag
> 
> I don't feel strongly either way on this.  It seemed useful at the time
> for being able to decouple upgrading libvirt and enabling features that
> come with that.  I'd like to let Dan get back from vacation and weigh in
> on it, though.

Yes, I think that version cap feature is valuable no matter what we
do about CI testing, which is why I +2'd it originally.

> 
> I wonder if the job could be as simple as one with an added step in the
> config to install latest libvirt from source.  Dan, do you think someone
> could add a libvirt-current.tar.gz to http://libvirt.org/sources/ ?
> Using the latest release seems better than master from git.

I'd strongly recommend against using GIT master. It will cause openstack
CI more pain than benefits. Using the latest stable release is a better
bet

> I'll mess around and see if I can spin up an experimental job.

There is work to add support for this in devestack already which I
prefer since it makes it easy for developers to get an environment
which matches the build system:

  https://review.openstack.org/#/c/108714/

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] Using host networking for docker containers

[Matthew Mosesohn]

Moving to host networking would reduce our ability to do zero downtime
upgrades in the future. It means you must kill the old container in
order to start the new one, rather than allowing for the possibility
to remap the network configuration in iptables. It's something we
don't have now, but we may be able to do in the future.

With regards to security issues, we have some more restrictive
firewall rules in place now. I don't think this is a major issue.

I don't think it makes a huge difference in performance to switch to
host networking, but it's worth testing.

On Mon, Aug 11, 2014 at 1:16 PM, Aleksandr Didenko
 wrote:
> Hi,
>
> we're running only 3 containers in privileged mode: cobbler, rsyslog and
> mcollective. Running all the containers in privileged mode is not a good
> idea for security reasons. Docker manages DNAT forwarding itself, so it does
> not create any overhead for us.
>
>
>> Is there any real benefits of using separate namespaces in security terms?
>
> Of course, for example only ports specified in EXPOSE line in Dockerfile are
> exposed to the host network. So if you start any additional tcp/udp
> listeners inside the containers, their ports won't be accessible from the
> host network.
>
>
>
> On Sat, Aug 9, 2014 at 10:39 AM, Dmitriy Shulyak 
> wrote:
>>
>> Hi team,
>>
>> I want to discuss benefits of using host networking [1] for docker
>> containers, on master node.
>>
>> This feature was added in docker 0.11 and basicly means - reuse host
>> networking stack, without
>> creating separate namespace for each container.
>>
>> In my opinion it will result in much more stable install/upgrade of master
>> node.
>>
>> 1. There will be no need for dhcrelay/dhcrelay_monitor on host
>> 2. No dnat port forwarding
>> 3. Performance improvement for pxe boot ???
>>
>> Is there any real benefits of using separate namespaces in security terms?
>>
>> To implement this we will need:
>>
>> 1. Update docker to recent version 0.12/1.x, we will do it anyway, yes?
>> 2. Run docker containers with --net=host
>>
>> Ofcourse it will require running containers in privileged mode, but afaik
>> we are already doing this for other reasons.
>>
>> So, what do you think?
>>
>> [1] https://github.com/docker/docker/issues/2012
>> [2] https://docs.docker.com/articles/networking/
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Gary Kotton]

In the past there was smokestack that used a fedora version of libvirt.
Any idea what happened to that?

On 8/11/14, 12:53 PM, "Daniel P. Berrange"  wrote:

>On Fri, Aug 08, 2014 at 09:06:29AM -0400, Russell Bryant wrote:
>> On 08/07/2014 08:06 PM, Michael Still wrote:
>> > It seems to me that the tension here is that there are groups who
>> > would really like to use features in newer libvirts that we don't CI
>> > on in the gate. Is it naive to think that a possible solution here is
>> > to do the following:
>> > 
>> >  - revert the libvirt version_cap flag
>> 
>> I don't feel strongly either way on this.  It seemed useful at the time
>> for being able to decouple upgrading libvirt and enabling features that
>> come with that.  I'd like to let Dan get back from vacation and weigh in
>> on it, though.
>
>Yes, I think that version cap feature is valuable no matter what we
>do about CI testing, which is why I +2'd it originally.
>
>> 
>> I wonder if the job could be as simple as one with an added step in the
>> config to install latest libvirt from source.  Dan, do you think someone
>> could add a libvirt-current.tar.gz to
>>https://urldefense.proofpoint.com/v1/url?u=http://libvirt.org/sources/&k=
>>oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPh
>>CZFxPEq8%3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=0298
>>f3506d0ac41e81fce6b0b3096455c3e78ba578970e42290997a766b0811e ?
>> Using the latest release seems better than master from git.
>
>I'd strongly recommend against using GIT master. It will cause openstack
>CI more pain than benefits. Using the latest stable release is a better
>bet
>
>> I'll mess around and see if I can spin up an experimental job.
>
>There is work to add support for this in devestack already which I
>prefer since it makes it easy for developers to get an environment
>which matches the build system:
>
>  https://review.openstack.org/#/c/108714/
>
>Regards,
>Daniel
>-- 
>|: 
>https://urldefense.proofpoint.com/v1/url?u=http://berrange.com/&k=oIvRg1%2
>BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%
>3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=d7e4cac70435cd
>1066f4f7ec8795f16c345c0ca4529bb4b3033762552817e775  -o-
>https://urldefense.proofpoint.com/v1/url?u=http://www.flickr.com/photos/db
>errange/&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfD
>tysg45MkPhCZFxPEq8%3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%
>0A&s=0f30863dcc9a34ffe520b0cee787f76bb67334963ebcd57d9b53b3890f0265ea :|
>|: 
>https://urldefense.proofpoint.com/v1/url?u=http://libvirt.org/&k=oIvRg1%2B
>dGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3
>D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=07044ea24f9483f
>b87f1c67511c4c62a3e2b8971ad8418b7a0a61068f623f529  -o-
> 
>https://urldefense.proofpoint.com/v1/url?u=http://virt-manager.org/&k=oIvR
>g1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxP
>Eq8%3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=185d42ae1a
>5b0422c1fa3306e55397461140024c38530f3a6986a72c51edec08 :|
>|: 
>https://urldefense.proofpoint.com/v1/url?u=http://autobuild.org/&k=oIvRg1%
>2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8
>%3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=2e295d841588d
>8cb243a6c37eb0c17007760ba0506cd621fff34de6a6f7c7b03   -o-
>https://urldefense.proofpoint.com/v1/url?u=http://search.cpan.org/~danberr
>/&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45M
>kPhCZFxPEq8%3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=39
>b473e1829f6e6c4f7bd4e27304334df8c0d1e9ac2cf7432490ac72fbe111e6 :|
>|: 
>https://urldefense.proofpoint.com/v1/url?u=http://entangle-photo.org/&k=oI
>vRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZF
>xPEq8%3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=c100dc7d
>e638a7bd0477a6d9723599041f442e80207213597a220e1a2cd34c74   -o-
>https://urldefense.proofpoint.com/v1/url?u=http://live.gnome.org/gtk-vnc&k
>=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPh
>CZFxPEq8%3D%0A&m=rPfYVc630ymUlJf3W60dbNbCgcW5TimMFx1ooqIdr38%3D%0A&s=5c43d
>0b2c8ad31d53813420ce9ab1885943b4ef88458006edcb2295f47c46846 :|
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Vijay Venkatachalam]

Hi:

Continuing from last week's LBaaS meeting...

Currently an entity cannot be sent to driver unless it is linked to 
loadbalancer because loadbalancer is the root object and driver information is 
only available with loadbalancer.

The request to the driver is delayed because of which error propagation becomes 
tricky.

Let's say a monitor was configured with timeout > delay there would be no error 
then.
When a listener is configured there will be a monitor creation/deployment error 
like "timeout configured greater than delay".

Unless the error is very clearly crafted the user won't be able to understand 
the error.

I am half-heartedly OK with current approach.

But, I would prefer Brandon's Solution - make provider an attribute in each of 
the entities to get rid of this problem.

What do others think?

Thanks,
Vijay V.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Russell Bryant]

On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
> There is work to add support for this in devestack already which I
> prefer since it makes it easy for developers to get an environment
> which matches the build system:
> 
>   https://review.openstack.org/#/c/108714/

Ah, cool.  Devstack is indeed a better place to put the build scripting.
 So, I think we should:

1) Get the above patch working, and then merged.

2) Get an experimental job going to use the above while we work on #3

3) Before the job can move into the check queue and potentially become
voting, it needs to not rely on downloading the source on every run.
IIRC, we can have nodepool build an image to use for these jobs that
includes the bits already installed.

I'll switch my efforts over to helping get the above completed.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Daniel P. Berrange]

On Mon, Aug 11, 2014 at 07:58:41AM -0400, Russell Bryant wrote:
> On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
> > There is work to add support for this in devestack already which I
> > prefer since it makes it easy for developers to get an environment
> > which matches the build system:
> > 
> >   https://review.openstack.org/#/c/108714/
> 
> Ah, cool.  Devstack is indeed a better place to put the build scripting.
>  So, I think we should:
> 
> 1) Get the above patch working, and then merged.
> 
> 2) Get an experimental job going to use the above while we work on #3
> 
> 3) Before the job can move into the check queue and potentially become
> voting, it needs to not rely on downloading the source on every run.

Don't we have the ability to mirror downloads locally to the build
system for python ?  The proposed patch allows an alternate download
URL to be set via an env variable so it could point to a local mirror
instead of libvirt.org / qemu.org

> IIRC, we can have nodepool build an image to use for these jobs that
> includes the bits already installed.


Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Russell Bryant]

On 08/09/2014 12:33 PM, Jeremy Stanley wrote:
> On 2014-08-08 09:06:29 -0400 (-0400), Russell Bryant wrote:
> [...]
>> We've seen several times that building and maintaining 3rd party
>> CI is a *lot* of work.
> 
> Building and maintaining *any* CI is a *lot* of work, not the least
> of which is the official OpenStack project CI (I believe Monty
> mentioned in #openstack-infra last night that our CI is about twice
> the size of Travis-CI now, not sure what metric he's comparing there
> though).

Dang, I'd love to see those numbers.  :-)

>> Like you said in [1], doing this in infra's CI would be ideal. I
>> think 3rd party should be reserved for when running it in the
>> project's infrastructure is not an option for some reason
>> (requires proprietary hw or sw, for example).
> 
> Add to the "not an option for some reason" list, software which is
> not easily obtainable through typical installation channels (PyPI,
> Linux distro-managed package repositories for their LTS/server
> releases, et cetera) or which requires gyrations which destabilize
> or significantly complicate maintenance of the overall system as
> well as reproducibility for developers. It may be possible to work
> around some of these concerns via access from multiple locations
> coupled with heavy caching, but adding that in for a one-off source
> is hard to justify the additional complexity too.

Understood.  Some questions ... is building an image that has libvirt
and qemu pre-installed from source good enough?  It avoids the
dependency on job runs, but moves it to image build time though, so it
still exists.

If the above still doesn't seem like a workable setup, then I think we
should just go straight to an image with fedora + virt-preview repo,
which kind of sounds easier, anyway.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Russell Bryant]

On 08/11/2014 07:58 AM, Russell Bryant wrote:
> On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
>> There is work to add support for this in devestack already which I
>> prefer since it makes it easy for developers to get an environment
>> which matches the build system:
>>
>>   https://review.openstack.org/#/c/108714/
> 
> Ah, cool.  Devstack is indeed a better place to put the build scripting.
>  So, I think we should:
> 
> 1) Get the above patch working, and then merged.
> 
> 2) Get an experimental job going to use the above while we work on #3
> 
> 3) Before the job can move into the check queue and potentially become
> voting, it needs to not rely on downloading the source on every run.
> IIRC, we can have nodepool build an image to use for these jobs that
> includes the bits already installed.
> 
> I'll switch my efforts over to helping get the above completed.
> 

I still think the devstack patch is good, but after some more thought, I
think a better long term CI job setup would just be a fedora image with
the virt-preview repo.  I think I'll try that ...

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Russell Bryant]

On 08/11/2014 08:01 AM, Daniel P. Berrange wrote:
> On Mon, Aug 11, 2014 at 07:58:41AM -0400, Russell Bryant wrote:
>> On 08/11/2014 05:53 AM, Daniel P. Berrange wrote:
>>> There is work to add support for this in devestack already which I
>>> prefer since it makes it easy for developers to get an environment
>>> which matches the build system:
>>>
>>>   https://review.openstack.org/#/c/108714/
>>
>> Ah, cool.  Devstack is indeed a better place to put the build scripting.
>>  So, I think we should:
>>
>> 1) Get the above patch working, and then merged.
>>
>> 2) Get an experimental job going to use the above while we work on #3
>>
>> 3) Before the job can move into the check queue and potentially become
>> voting, it needs to not rely on downloading the source on every run.
> 
> Don't we have the ability to mirror downloads locally to the build
> system for python ?  The proposed patch allows an alternate download
> URL to be set via an env variable so it could point to a local mirror
> instead of libvirt.org / qemu.org

There's a pypi mirror at least.  I'm not sure about mirroring other things.

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] The future of the integrated release

[Jeremy Stanley]

On 2014-08-11 11:02:52 +0200 (+0200), Thierry Carrez wrote:
> Ideally we'd want a blocking vote that carries over patchsets, but which
> can be set or unset by any core/driver/ptl (to be determined). So if the
> PTL sets the "procedural" Workflow-2 at feature freeze, I can clear it
> when RC1 is published.
> 
> Not sure that's supported in Gerrit though :)

As far as I know it's not... otherwise we'd have made WIP do the
same as you describe.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Nova] PCI support

[Gary Kotton]

Hi,
At the moment all of the drivers are required CI support. Are there any plans 
regarding the PIC support. I understand that this is something that requires 
specific hardware. Are there any plans to add this?
Thanks
Gary
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: FW: [Neutron] Group Based Policy and the way forward

[CARVER, PAUL]

loy wolfe [mailto:loywo...@gmail.com] wrote:

>Then since Network/Subnet/Port will never be treated just as LEGACY
>COMPATIBLE role, there is no need to extend Nova-Neutron interface to
>follow the GBP resource. Anyway, one of optional service plugins inside
>Neutron shouldn't has any impact on Nova side.

This gets to the root of why I was getting confused about Jay and others
having Nova related concerns. I was/am assuming that GBP is simply an
*additional* mechanism for manipulating Neutron, not a deprecation of any
part of the existing Neutron API. I think Jay's concern and the reason
why he keeps mentioning Nova as the biggest and most important consumer
of Neutron's API stems from an assumption that Nova would need to change
to use the GBP API.

If I've understood the follow on discussions correctly, there's no need for
Nova to use the GBP API at all until/unless the Nova developers see benefit
in it because they can continue to accomplish everything with the existing
API. The GBP API simply provides a more application centric rather than
network centric representation of the same thing.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Jeremy Stanley]

On 2014-08-11 08:04:34 -0400 (-0400), Russell Bryant wrote:
> Dang, I'd love to see those numbers.  :-)

Me too. Now that I'm not travelling I'll see if I can find out what
he meant by that.

> Understood.  Some questions ... is building an image that has libvirt
> and qemu pre-installed from source good enough?  It avoids the
> dependency on job runs, but moves it to image build time though, so it
> still exists.

Moving complex stability risks to image creation time still causes
us to potentially fail to update our worker images as often, which
means tests randomly run on increasingly stale systems in some
providers/regions until the issue is noticed, identified and
addressed. That said, we do already compile some things during job
runs today (in particular, library bindings which get install-time
linked by some Python modules).

In reality, depending on more things gathered from different places
on the Internet (be it Git repository sites like GitHub/Bitbucket,
or private package collections) decreases our overall stability far
more than compiling things does.

> If the above still doesn't seem like a workable setup, then I think we
> should just go straight to an image with fedora + virt-preview repo,
> which kind of sounds easier, anyway.

If it's published from EPEL or whatever Fedora's equivalent is, then
that's probably fine. If it's served from a separate site, then that
increases the chances that we run into network issues either at
image build time or job run time. Also, we would want to make sure
whatever solution we settle on is well integrated within DevStack
itself, so that individual developers can recreate these conditions
themselves without a lot of additional work.

One other thing to keep in mind... Fedora's lifecycle is too short
for us to support outside of jobs for our master branches, so this
would not be a solution beyond release time (we couldn't continue to
run these jobs for Juno once released if the solution hinges on
Fedora). Getting the versions we want developers and deployers to
use into Ubuntu 14.04 Cloud Archive and CentOS (RHEL) 7 EPEL on the
other hand would be a much more viable long-term solution.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [keystone] how to generate keystone.conf.sample in havana release?

[Dolph Mathews]

Unfortunately, there is no way to generate a sample conf in the Havana
release. The etc/keystone.conf.sample included in Havana was maintained
manually. In icehouse (and master), you can generate a sample configuration
file using tox:

  $ tox -e sample_config

Or run generate_sample.sh directly if you already have all the dependencies
available:

  $ sh tools/config/generate_sample.sh

-- Forwarded message --
From: jazeltq 
Date: Mon, Aug 11, 2014 at 5:06 AM
Subject: how to generate keystone.conf.sample in havana release?
To: Dolph Mathews 


Hello
Is there a way to auto-generate keystone.conf.sample in havana release
keystone?
--
This message was sent from Launchpad by
jazeltq (https://launchpad.net/~jazeltq-k)
using the "Contact this team's admins" link on the Keystone Drivers team
page
(https://launchpad.net/~keystone-drivers).
For more information see
https://help.launchpad.net/YourAccount/ContactingPeople
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [sahara] Backports for 2014.1.2

[Sergey Lukjanov]

There were no proposals for additional backports for sahara 2014.1.2,
so, I'm going to release it later today.

On Thu, Aug 7, 2014 at 10:48 PM, Sergey Lukjanov  wrote:
> Hey sahara folks,
>
> I'm going to push 2014.1.2 tag to stable/icehouse branch next week,
> so, please, propose backports before the weekend and ping us to
> backport some sensitive fixes.
>
> Thanks you!
> --
> Sincerely yours,
> Sergey Lukjanov
> Sahara Technical Lead
> (OpenStack Data Processing)
> Principal Software Engineer
> Mirantis Inc.



-- 
Sincerely yours,
Sergey Lukjanov
Sahara Technical Lead
(OpenStack Data Processing)
Principal Software Engineer
Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Russell Bryant]

On 08/11/2014 09:17 AM, Jeremy Stanley wrote:
> On 2014-08-11 08:04:34 -0400 (-0400), Russell Bryant wrote:
>> Dang, I'd love to see those numbers.  :-)
> 
> Me too. Now that I'm not travelling I'll see if I can find out what
> he meant by that.
> 
>> Understood.  Some questions ... is building an image that has libvirt
>> and qemu pre-installed from source good enough?  It avoids the
>> dependency on job runs, but moves it to image build time though, so it
>> still exists.
> 
> Moving complex stability risks to image creation time still causes
> us to potentially fail to update our worker images as often, which
> means tests randomly run on increasingly stale systems in some
> providers/regions until the issue is noticed, identified and
> addressed. That said, we do already compile some things during job
> runs today (in particular, library bindings which get install-time
> linked by some Python modules).
> 
> In reality, depending on more things gathered from different places
> on the Internet (be it Git repository sites like GitHub/Bitbucket,
> or private package collections) decreases our overall stability far
> more than compiling things does.
> 
>> If the above still doesn't seem like a workable setup, then I think we
>> should just go straight to an image with fedora + virt-preview repo,
>> which kind of sounds easier, anyway.
> 
> If it's published from EPEL or whatever Fedora's equivalent is, then
> that's probably fine. If it's served from a separate site, then that
> increases the chances that we run into network issues either at
> image build time or job run time. Also, we would want to make sure
> whatever solution we settle on is well integrated within DevStack
> itself, so that individual developers can recreate these conditions
> themselves without a lot of additional work.

EPEL is a repo produced by the Fedora project for RHEL and its
derivatives.  The virt-preview repo is hosted on fedorapeople.org, which
is where custom repos live.  I'd say it's more analogous to Ubuntu's PPAs.

https://fedorapeople.org/groups/virt/virt-preview/

> One other thing to keep in mind... Fedora's lifecycle is too short
> for us to support outside of jobs for our master branches, so this
> would not be a solution beyond release time (we couldn't continue to
> run these jobs for Juno once released if the solution hinges on
> Fedora). Getting the versions we want developers and deployers to
> use into Ubuntu 14.04 Cloud Archive and CentOS (RHEL) 7 EPEL on the
> other hand would be a much more viable long-term solution.

Yep, makes sense.

For testing bleeding edge, I've also got my eye on how we could do this
with CentOS.  There is a virt SIG in CentOS that I'm hoping will produce
something similar to Fedora's virt-preview repo, but it's not there yet.
 I'm going to go off and discuss this with the SIG there.

http://wiki.centos.org/SpecialInterestGroup/Virtualization

-- 
Russell Bryant

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [sahara] new client release (0.7.1)

[Sergey Lukjanov]

Hey sahara folks,

the latest sahara client release was at Mar 29 (right after the fully
completed renaming process) and so we have a lot of unreleased fixes
and improvements.

You can check diff [0] and launchpad's release page [1].

I'm going to release 0.7.1 this week, so, please propose changes to be
included into the new client release in the next few days.

Thanks.

[0] https://github.com/openstack/python-saharaclient/compare/0.7.0...HEAD
[1] https://launchpad.net/python-saharaclient/+milestone/0.7.1

-- 
Sincerely yours,
Sergey Lukjanov
Sahara Technical Lead
(OpenStack Data Processing)
Principal Software Engineer
Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Jeremy Stanley]

On 2014-08-11 08:07:02 -0400 (-0400), Russell Bryant wrote:
> There's a pypi mirror at least.  I'm not sure about mirroring
> other things.

Right, that's a specific solution for mirroring the pypi.python.org
cheeseshop. We've got our (Infra) sights set on mirroring Ubuntu and
CentOS package repositories to similarly reduce the incidence of job
run-time failures we see getting updated packages and indexes from
distro sites and cloud provider mirrors, but those too are
nontrivial efforts and will be a while yet before we have them in
place. Mirroring things is generally complex, since different kinds
of files/data need widely differing retrieval, indexing and caching
solutions--there's no one-size-fits-all option really.

Perhaps another good example is the Fedora qcow2 image we download
and cache on DevStack worker images so that Heat can perform some of
its more complex integration tests... failures encountered when
obtaining that image from dl.fedoraproject.org are (last time I
checked anyway) our most frequent cause of nodepool update problems.
We could set up our own mirror of that file of course, but to some
extent that's still just moving the problem--each additional
mirroring solution is something new we have to monitor, maintain and
troubleshoot so we must ask ourselves whether the increased
management burden from that new complexity is balanced by potential
decreases in management burden found by improving stability in other
parts of the system.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [designate] [neutron] designate and neutron integration

[Miyashita, Kazuhiro]

Hi,

I want to ask about neutron and designate integration.
I think dnsmasq fowards DNS request from instance to designate is better.

   ++
   |DNS server(designate)   |
   ++
|
-+--+-- Network1
 |
  ++
  |dnsmasq |
  ++
|
-+--+-- Network2
 |
+-+
|instance |
+-+

Because it's simpler than virtual router connects Network1 and Network2.
If router connects network, instance should know where DNS server is. it is 
complicated.
dnsmasq returns its ip address as dns server in DHCP replay by ordinary, so,
I think good dnsmasq becomes like a gateway to designate.

But, I can't connect dnsmasq to Network1. because of today's neutron design.

Question:
  Does designate design team have a plan such as above integration?
  or other integration design?

*1: Network1 and Network2 are deployed by neutron.
*2: neutron deploys dnsmasq as a dhcp server.
dnsmasq can forward DNS request.

Thanks,

kazuhiro MIYASHITA




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [OSSN 0022] Nova Networking does not enforce security group rules following a soft reboot of an instance

[Nathan Kinder]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Nova Networking does not enforce security group rules following a soft
reboot of an instance
- ---

### Summary ###
In deployments using Nova Networking, security group rules associated
with an instance may not be enforced after a soft reboot. Nova is
designed to apply the configured security group rules to an instance
when certain operations are performed, such as a normal boot operation.
If an operation has been performed that results in the clearing of
security group rules, such as restarting the nova compute service, then
performing a soft reboot of that instance will cause it to be
started without security group rules being applied.

Deployments using Neutron are not impacted.

### Affected Services / Software ###
Nova, Havana, Grizzly

### Discussion ###
In Nova deployments using Nova Networking, security groups are
implemented using iptables, which is used to configure and control
network traffic into Nova instances. When an instance is first booted
using the normal boot method (nova boot ), the security
group rules are applied to that instance.

When an instance is rebooted using the soft reboot method (nova reboot
), the security group rules are not reapplied since they
should have been already applied when the instance was initially
booted. If the security group rules have not been applied following an
event that resulted in their clearing, such as restarting the compute
service, the instance will be brought up without security group
enforcement. This situation is most likely to arise in cases where the
Nova compute service has been terminated or restarted, which removes
all iptables rules. If a stopped instance is then started by using a
soft reboot, it will not have any security group rules applied. A hard
reboot (nova reboot --hard ) reapplies the security group
rules, so it is not susceptible to this issue.

Depending on the deployment architecture, this could breach security
assumptions and leave an instance vulnerable to network based attacks.

This issue only affects the Havana and Grizzly releases. The Icehouse
release does not allow a stopped instance to be started using a soft
reboot, therefore this issue does not affect the Icehouse release.

### Recommended Actions ###
Do not to use the soft reboot method to start instances from the
stopped state. If instances are in the stopped state, boot using "nova
boot " or reboot using "nova reboot --hard "
to force the security group rules to be applied.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0022
Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/1316822
OpenStack Security ML : openstack-secur...@lists.openstack.org
OpenStack Security Group : https://launchpad.net/~openstack-ossg
-BEGIN PGP SIGNATURE-
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJT6MtUAAoJEJa+6E7Ri+EV2t4H/RwM5auF4ik9tdZuRNVLI/Rv
Si1SBx+EZA5QOksJ1r476mm2nA9+OnhCZRVnz8bHGiKtzC7eRV82c1OO1q0R/w1/
zHuTd6ZsZaAyBshfv6YoEVoRn6I2duYiR6gRHz/+hfrItt5E+SYBXqFskJazJ6dF
PZBA16qrIPmeir1eDpDvFnpbMEkAMUDKGZNba/DXwEZgdKVdFaWPI3cNIsHVP6bL
fsqMX+7bldB8kBDKfyFleFYqBUI5Anonzrq1tu6TS0vXl4gsfD2LE5LphPTYaiC+
EMypsACSafd19RhpeHS7/I1iVhf/Xa26Bvne0T8HWMLuDd1M5TT9E7nnas45k24=
=1x8I
-END PGP SIGNATURE-

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Daniel P. Berrange]

On Mon, Aug 11, 2014 at 01:45:56PM +, Jeremy Stanley wrote:
> On 2014-08-11 08:07:02 -0400 (-0400), Russell Bryant wrote:
> > There's a pypi mirror at least.  I'm not sure about mirroring
> > other things.
> 
> Right, that's a specific solution for mirroring the pypi.python.org
> cheeseshop. We've got our (Infra) sights set on mirroring Ubuntu and
> CentOS package repositories to similarly reduce the incidence of job
> run-time failures we see getting updated packages and indexes from
> distro sites and cloud provider mirrors, but those too are
> nontrivial efforts and will be a while yet before we have them in
> place. Mirroring things is generally complex, since different kinds
> of files/data need widely differing retrieval, indexing and caching
> solutions--there's no one-size-fits-all option really.

If there are specific things we could do to libvirt.org / qemu.org
download sites, to make mirroring easier or more reliable for OpenStack,
we could certainly explore options there, since we know the right people
involved in both projects. 

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Samuel Bercovici]

Hi,

Validations such as "timeout > delay" should be performed on the API level 
before it reaches the driver.

For a configuration tree (lb, listeners, pools, etc.), there should be one 
provider.
Having provider defined in multiple places does not make sense.


-San.


From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]
Sent: Monday, August 11, 2014 2:43 PM
To: OpenStack Development Mailing List (openstack-dev@lists.openstack.org)
Subject: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver 
interface on every API request"

Hi:

Continuing from last week's LBaaS meeting...

Currently an entity cannot be sent to driver unless it is linked to 
loadbalancer because loadbalancer is the root object and driver information is 
only available with loadbalancer.

The request to the driver is delayed because of which error propagation becomes 
tricky.

Let's say a monitor was configured with timeout > delay there would be no error 
then.
When a listener is configured there will be a monitor creation/deployment error 
like "timeout configured greater than delay".

Unless the error is very clearly crafted the user won't be able to understand 
the error.

I am half-heartedly OK with current approach.

But, I would prefer Brandon's Solution - make provider an attribute in each of 
the entities to get rid of this problem.

What do others think?

Thanks,
Vijay V.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] fair standards for all hypervisor drivers

[Jeremy Stanley]

On 2014-08-11 14:56:27 +0100 (+0100), Daniel P. Berrange wrote:
> If there are specific things we could do to libvirt.org / qemu.org
> download sites, to make mirroring easier or more reliable for
> OpenStack, we could certainly explore options there, since we know
> the right people involved in both projects.

Mirroring is generally a reaction to stability issues more than
anything. Providing those files from multiple locations behind a
global cache/load-balancing solution (maybe a CDN) to ensure
stability would likely help prevent us from needing to have yet
another one-off mirror for a handful of files. It's worth adding to
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
and discussing tomorrow, if you're around, so we can be sure to
get input from more of the Infra team.
-- 
Jeremy Stanley

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] how to generate conf sample in havana?

[Dolph Mathews]

See here:
http://lists.openstack.org/pipermail/openstack-dev/2014-August/042573.html


On Mon, Aug 11, 2014 at 3:53 AM, Li Tianqing  wrote:

> Hello,
>Right now, i customerized keystone to meet our requeriments. I want to
> add some configuration options in keystone.conf.sample. But, i
> do not find generate_sample.sh in directory tools in Havana keystone. So i
> want to know how to generate conf sample in havana keystone.
> I know, i can write directly to keystone.conf.sample, but should there be
> something like generate_sample.sh in keystone?
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] PCI support

[Robert Li (baoli)]

Gary,

Cisco is adding it in our CI testbed. I guess that mlnx is doing the same for 
their MD as well.

—Robert

On 8/11/14, 9:05 AM, "Gary Kotton" 
mailto:gkot...@vmware.com>> wrote:

Hi,
At the moment all of the drivers are required CI support. Are there any plans 
regarding the PIC support. I understand that this is something that requires 
specific hardware. Are there any plans to add this?
Thanks
Gary
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Is network ordering of vNICs guaranteed?

[CARVER, PAUL]

Armando M. [mailto:arma...@gmail.com] wrote:

>>On 9 August 2014 10:16, Jay Pipes  wrote:
>>Paul, does this friend of a friend have a reproduceable test
>>script for this?

>We would also need to know the OpenStack release where this issue manifest
>itself. A number of bugs have been raised in the past around this type of
>issue, and the last fix I recall is this one:
>
>https://bugs.launchpad.net/nova/+bug/1300325
>
>It's possible that this might have regressed, though.

The reason I called it "friend of a friend" is because I think the info
has filtered through a series of people and is not firsthand observation.
I'll ask them to track back to who actually observed the behavior, how
long ago, and with what version.

It could be a regression, or it could just be old info that people have
continued to assume is true without realizing it was considered a bug
all along and has been fixed.

Thanks! The moment I first heard it my first reaction was that it was
almost certainly a bug and had probably already been fixed.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] how to generate keystone.conf.sample in havana release?

[Li Tianqing]

Ok， thank you all the same.

At 2014-08-11 09:31:19, "Dolph Mathews"  wrote:

Unfortunately, there is no way to generate a sample conf in the Havana release. 
The etc/keystone.conf.sample included in Havana was maintained manually. In 
icehouse (and master), you can generate a sample configuration file using tox:


  $ tox -e sample_config


Or run generate_sample.sh directly if you already have all the dependencies 
available:


  $ sh tools/config/generate_sample.sh


-- Forwarded message --
From: jazeltq
Date: Mon, Aug 11, 2014 at 5:06 AM
Subject: how to generate keystone.conf.sample in havana release?
To: Dolph Mathews 


Hello
Is there a way to auto-generate keystone.conf.sample in havana release
keystone?
--
This message was sent from Launchpad by
jazeltq (https://launchpad.net/~jazeltq-k)
using the "Contact this team's admins" link on the Keystone Drivers team page
(https://launchpad.net/~keystone-drivers).
For more information see
https://help.launchpad.net/YourAccount/ContactingPeople


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] PCI support

[Gary Kotton]

Thanks for the update.

From: "Robert Li (baoli)" mailto:ba...@cisco.com>>
Reply-To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, August 11, 2014 at 5:08 PM
To: OpenStack List 
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [Nova] PCI support

Gary,

Cisco is adding it in our CI testbed. I guess that mlnx is doing the same for 
their MD as well.

-Robert

On 8/11/14, 9:05 AM, "Gary Kotton" 
mailto:gkot...@vmware.com>> wrote:

Hi,
At the moment all of the drivers are required CI support. Are there any plans 
regarding the PIC support. I understand that this is something that requires 
specific hardware. Are there any plans to add this?
Thanks
Gary
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

[Jay Pipes]

On 08/10/2014 10:36 PM, Jay Lau wrote:

I was asking this because I got a "-2" for
https://review.openstack.org/109505 , just want to know why this new
term "metadetails" was invented when we already have "details",
"metadata", "system_metadata", "instance_metadata", and "properties" (on
images and volumes).


As the person who -2'd the review, I'm thankful you raised this issue on 
the ML, Jay. Much appreciated.


Eagerly awaiting answers,
-jay

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

[Clint Byrum]

Excerpts from Steve Baker's message of 2014-08-10 15:33:26 -0700:
> On 02/08/14 04:07, Allison Randal wrote:
> > A few of us have been independently experimenting with Ansible as a
> > backend for TripleO, and have just decided to try experimenting
> > together. I've chatted with Robert, and he says that TripleO was always
> > intended to have pluggable backends (CM layer), and just never had
> > anyone interested in working on them. (I see it now, even in the early
> > docs and talks, I guess I just couldn't see the forest for the trees.)
> > So, the work is in line with the overall goals of the TripleO project.
> >
> > We're starting with a tiny scope, focused only on updating a running
> > TripleO deployment, so our first work is in:
> >
> > - Create an Ansible Dynamic Inventory plugin to extract metadata from Heat
> > - Improve/extend the Ansible nova_compute Cloud Module (or create a new
> > one), for Nova rebuild
> > - Develop a minimal handoff from Heat to Ansible, particularly focused
> > on the interactions between os-collect-config and Ansible
> >
> > We're merging our work in this repo, until we figure out where it should
> > live:
> >
> > https://github.com/allisonrandal/tripleo-ansible
> >
> > We've set ourselves one week as the first sanity-check to see whether
> > this idea is going anywhere, and we may scrap it all at that point. But,
> > it seems best to be totally transparent about the idea from the start,
> > so no-one is surprised later.
> >
> Having pluggable backends for configuration seems like a good idea, and
> Ansible is a great choice for the first alternative backend.
> 

TripleO is intended to be loosely coupled for many components, not just
in-instance configuration.

> However what this repo seems to be doing at the moment is bypassing heat
> to do a stack update, and I can only assume there is an eventual goal to
> not use heat at all for stack orchestration too.
>
>
> Granted, until blueprint update-failure-recovery lands[1] then doing a
> stack-update is about as much fun as russian roulette. But this effort
> is tactical rather than strategic, especially given TripleO's mission
> statement.
> 

We intend to stay modular. Ansible won't replace Heat from end to end.

Right now we're stuck with an update that just doesn't work. It isn't
just about update-failure-recovery, which is coming along nicely, but
it is also about the lack of signals to control rebuild, poor support
for addressing machines as groups, and unacceptable performance in
large stacks.

We remain committed to driving these things into Heat, which will allow
us to address these things the way a large scale operation will need to.

But until we can land those things in Heat, we need something more
flexible like Ansible to go around Heat and do things in the exact
order we need them done. Ansible doesn't have a REST API, which is a
non-starter for modern automation, but the need to control workflow is
greater than the need to have a REST API at this point.

> If I were to use Ansible for TripleO configuration I would start with
> something like the following:
> * Install an ansible software-config hook onto the image to be triggered
> by os-refresh-config[2][3]
> * Incrementally replace StructuredConfig resources in
> tripleo-heat-templates with SoftwareConfig resources that include the
> ansible playbooks via get_file
> * The above can start in a fork of tripleo-heat-templates, but can
> eventually be structured using resource providers so that the deployer
> chooses what configuration backend to use by selecting the environment
> file that contains the appropriate config resources
> 
> Now you have a cloud orchestrated by heat and configured by Ansible. If
> it is still deemed necessary to do an out-of-band update to the stack
> then you're in a much better position to do an ansible push, since you
> can use the same playbook files that heat used to bring up the stack.
> 

That would be a good plan if we wanted to fix issues with os-*-config,
but that is the opposite of reality. We are working around Heat
orchestration issues with Ansible.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Is network ordering of vNICs guaranteed?

[Jay Pipes]

Thanks, Paul!

On 08/11/2014 10:10 AM, CARVER, PAUL wrote:

Armando M. [mailto:arma...@gmail.com] wrote:


On 9 August 2014 10:16, Jay Pipes  wrote:



Paul, does this friend of a friend have a reproduceable test



script for this?



We would also need to know the OpenStack release where this issue manifest



itself. A number of bugs have been raised in the past around this type of



issue, and the last fix I recall is this one:







https://bugs.launchpad.net/nova/+bug/1300325







It's possible that this might have regressed, though.


The reason I called it "friend of a friend" is because I think the info

has filtered through a series of people and is not firsthand observation.

I'll ask them to track back to who actually observed the behavior, how

long ago, and with what version.

It could be a regression, or it could just be old info that people have

continued to assume is true without realizing it was considered a bug

all along and has been fixed.

Thanks! The moment I first heard it my first reaction was that it was

almost certainly a bug and had probably already been fixed.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

[Dan Smith]

> As the person who -2'd the review, I'm thankful you raised this issue on
> the ML, Jay. Much appreciated.

The "metadetails" term isn't being invented in this patch, of course. I
originally complained about the difference when this was being added:

https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contrib/server_groups.py,cm

As best I can tell, the response in that patch set about why it's being
translated is wrong (backwards). I expect that the API extension at the
time called it "metadetails" and they decided to make the object the
same and do the translation there.

From what I can tell, the actual server_group API extension that made it
into the tree never got the ability to set/change/etc the
metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
wrongly.

If we care to have this functionality, then I propose we change the
attribute on the object (we can handle this with versioning) and reflect
it as "metadata" in the API.

However, I have to ask: do we really need another distinct metadata
store attached to server_groups? If not, how about we just remove it
from the database and the object, clean up the bit of residue that is
still in the API extension and be done with it?

--Dan



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

[Jay Pipes]

On 08/11/2014 11:06 AM, Dan Smith wrote:

As the person who -2'd the review, I'm thankful you raised this issue on
the ML, Jay. Much appreciated.


The "metadetails" term isn't being invented in this patch, of course. I
originally complained about the difference when this was being added:

https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contrib/server_groups.py,cm

As best I can tell, the response in that patch set about why it's being
translated is wrong (backwards). I expect that the API extension at the
time called it "metadetails" and they decided to make the object the
same and do the translation there.

 From what I can tell, the actual server_group API extension that made it
into the tree never got the ability to set/change/etc the
metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
wrongly.

If we care to have this functionality, then I propose we change the
attribute on the object (we can handle this with versioning) and reflect
it as "metadata" in the API.

However, I have to ask: do we really need another distinct metadata
store attached to server_groups?


No.

> If not, how about we just remove it

from the database and the object, clean up the bit of residue that is
still in the API extension and be done with it?


+1

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

[Zane Bitter]

On 11/08/14 10:46, Clint Byrum wrote:

Right now we're stuck with an update that just doesn't work. It isn't
just about update-failure-recovery, which is coming along nicely, but
it is also about the lack of signals to control rebuild, poor support
for addressing machines as groups, and unacceptable performance in
large stacks.


Are there blueprints/bugs filed for all of these issues?

-ZB

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree

[Robert Kukura]

On 8/11/14, 4:52 AM, Thierry Carrez wrote:

gustavo panizzo (gfa) wrote:

only one think i didn't like it

why all url,api, etc has to include the word 'preview'?
i imagine that i would be consuming the new feature using heat, puppet,
local scripts, custom horizon, whatever. Why do you make me to change
all them when the feature moves out of preview? it could be a lot of
rework (for consumers) without gain. I would totally support other big
fat warnings everywhere (logs, documentation, startup log of
neutron-server) but don't change the API if is not necessary

I see two issues with this proposal: the first one is what Gustavo just
said: the use of the "preview" package/configoption/API creates friction
when the feature needs to go mainstream (package renaming, change in
configuration for deployers, change in API calls for users...).

Hi Thierry,

I completely agree with you and with Gustavo that "mangling" the REST 
URIs to include "preview" may have more cost (i.e. friction when the API 
becomes stable) than benefit. I'm happy to drop that particular part of 
the proposal. The email was intended to kick off discussion of these 
sorts of details.


My understanding is that the goal is to make it easy for people to "try"
the preview feature, and keeping the experimental feature in-tree is
seen as simpler to experiment with. But the pain from this friction imho
outweighs the pain of deploying an out-of-tree plugin for deployers.
I agree out-of-tree is a better option for truly experimental features. 
This in-tree stabilization is intended for a beta release, as opposed to 
a prototype.


The second issue is that once the feature is in "preview" in tree, it
moves the responsibility/burden of making it official (or removed) to
the core developers (as Salvatore mentioned). I kind of like the
approach where experimental features are developed in faster iterations
out-of-tree and when they are celebrated by experimenters and are ready
to be stable-supported, they are moved in tree.
I don't think we are really disagreeing here. There are clearly 
situations where rapid iteration out-of-tree, without the burden of the 
core review process, is most appropriate. But this proposal is intended 
for features that are on the cusp of being declared stable, rather than 
for experimentation. The intent is absolutely to have all changes to the 
code go through the regular core review process during this 
stabilization phase. This enables the feature to be fully reviewed and 
integrated (also including CLIs, Horizon and Heat support, 
documentation, etc.) at the point when the decision is made that no 
further incompatible API changes will be needed. Once the feature is 
declared stable, from the end-user perspective, its just a matter of 
removing the "preview" label. Moving the feature's code from the preview 
subtree to its normal locations in the tree will not effect most users 
or operators.


Note that the GBP team had implemented a proof-of-concept prior to the 
start of the Juno cycle out-of-tree. Our initial plan was to get this 
PoC code reviewed and merged at the beginning of Juno, and then 
iteratively improve it throughout the cycle. But we got a lot of 
resistance to the idea of merging a large body of code that had been 
developed outside the Neutron development and review process. We've 
instead had to break it into multiple pieces, and make sure each of 
those is production ready, to have any chance of getting through the 
review process during Juno.  Its not really clear that something 
significant developed externally can ever be "moved in tree", at least 
without major upheaval, including incompatible API changes, as it goes 
through the review/merge process.


Finally, consider that many interesting potential features for Neutron 
involve integrations with external back-ends, such as ODL or 
vendor-specific devices or controllers, along with a reference 
implementation that doesn't depend on external systems. To really 
validate that the API, the model, and the driver framework code for a 
new feature are all stable, it is necessary to implement and deploy 
several of these back-end integrations along with the reference 
implementation. But vendors may not be willing to invest substantially 
in this integration effort without the assurances about the quality and 
relative stability of the interfaces involved that comes from the core 
review process, and without the clear path to market that comes with 
in-tree development of an approved blueprint targeted at a specific 
Neutron release.


-Bob


Regards,




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Manila] Incubation request

[Swartzlander, Ben]

I just saw the agenda for tomorrow's TC meeting and we're on it. I plan to be 
there.

https://wiki.openstack.org/wiki/Meetings#Technical_Committee_meeting

-Ben


From: Swartzlander, Ben [mailto:ben.swartzlan...@netapp.com]
Sent: Monday, July 28, 2014 9:53 PM
To: openstack-dev@lists.openstack.org
Subject: [openstack-dev] [Manila] Incubation request

Manila has come a long way since we proposed it for incubation last autumn. 
Below are the formal requests.

https://wiki.openstack.org/wiki/Manila/Incubation_Application
https://wiki.openstack.org/wiki/Manila/Program_Application

Anyone have anything to add before I forward these to the TC?

-Ben Swartzlander

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Fuel] 6.0 blueprints cleanup

[Dmitry Pyzhov]

We've moved all blueprints from 6.0 to 'next' milestone. It has been done
in order to better view of stuff that we really want to implement in 6.0.

Feature freeze for 6.0 release is planned to 18th of September. If you are
going to merge your blueprint before that date, you can move it to 6.0
milestone and 6.0.x series. But blueprint must have fixed scope and must be
assigned to person who will lead this activity.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] 6.0 blueprints cleanup

[Dmitry Borodaenko]

All,

Please refer to this email for a list of information that has to be
populated in a blueprint before it can be assigned and scheduled:
http://lists.openstack.org/pipermail/openstack-dev/2014-August/042042.html

Please don't schedule a blueprint to 6.0 until it has these details
and its assignees are confirmed.

On Mon, Aug 11, 2014 at 8:30 AM, Dmitry Pyzhov  wrote:
> We've moved all blueprints from 6.0 to 'next' milestone. It has been done in
> order to better view of stuff that we really want to implement in 6.0.
>
> Feature freeze for 6.0 release is planned to 18th of September. If you are
> going to merge your blueprint before that date, you can move it to 6.0
> milestone and 6.0.x series. But blueprint must have fixed scope and must be
> assigned to person who will lead this activity.
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Dmitry Borodaenko

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Brandon Logan]

Yeah what I meant was the only solution I could come up with so that the
driver gets passed every call is to have the every entity have a
provider.  I do believe this is a bit cumbersome for a user, and extra
validation would be needed to verify that two entities linked together
cannot have different providers, but that would be pretty easy.  In my
opinion, it'd be a bit weird to have them all have a provider.  However,
there are some pros to it such as:

1) Driver always gets the create, update, and delete calls.
2) If drivers support a varying range of values for certain entity
attributes, that validation can be caught immediately if thats something
people wanted.
3) Will remove the necessity of a DEFERRED status for some drivers
(This also brings up a CON, in that some drivers may use DEFERRED and
some may not, which leads to an inconsistent UX).
4) Status management in some drivers will become a bit easier.

Still I don't think it is something that should be done because having
the user give a provider for every entity is a bit cumbersome.  Though
if enough people want this then a larger discussion about it should
probably happen.

Thanks,
Brandon

On Mon, 2014-08-11 at 13:57 +, Samuel Bercovici wrote:
> Hi,
> 
>  
> 
> Validations such as “timeout > delay” should be performed on the API
> level before it reaches the driver.
> 
>  
> 
> For a configuration tree (lb, listeners, pools, etc.), there should be
> one provider. 
> 
> Having provider defined in multiple places does not make sense.
> 
>  
> 
>  
> 
> -San.
> 
>  
> 
>  
> 
> From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com] 
> Sent: Monday, August 11, 2014 2:43 PM
> To: OpenStack Development Mailing List
> (openstack-dev@lists.openstack.org)
> Subject: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling
> driver interface on every API request"
> 
> 
>  
> 
> Hi:
> 
>  
> 
> Continuing from last week’s LBaaS meeting…
> 
>  
> 
> Currently an entity cannot be sent to driver unless it is linked to
> loadbalancer because loadbalancer is the root object and driver
> information is only available with loadbalancer. 
> 
>  
> 
> The request to the driver is delayed because of which error
> propagation becomes tricky.
> 
>  
> 
> Let’s say a monitor was configured with timeout > delay there would be
> no error then.
> 
> When a listener is configured there will be a monitor
> creation/deployment error like “timeout configured greater than
> delay”.
> 
>  
> 
> Unless the error is very clearly crafted the user won’t be able to
> understand the error.
> 
>  
> 
> I am half-heartedly OK with current approach. 
> 
>  
> 
> But, I would prefer Brandon’s Solution – make provider an attribute in
> each of the entities to get rid of this problem. 
> 
>  
> 
> What do others think?
> 
>  
> 
> Thanks,
> 
> Vijay V.
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] devstack local.conf file

[Nikesh Kumar Mahalka]

Hi,
I have gone through devstack links.
They are not clear like openstack.org documents.


For Example:
when i am using below local.conf file in devstack,"hp_msa_driver" is not
coming in "enabled_backends" in cinder.conf after running stack.sh.

[[local|localrc]]
ADMIN_PASSWORD=vedams123
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=ADMIN
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.2.170/29
HOST_IP=192.168.2.151
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
SYSLOG=True
SYSLOG_HOST=$HOST_IP
SYSLOG_PORT=516
TEMPEST_VOLUME_DRIVER=hp_msa_fc
TEMPEST_VOLUME_VENDOR="Hewlett-Packard"
TEMPEST_STORAGE_PROTOCOL=FC


[[post-config|$CINDER_CONF]]
[hp_msa_driver]
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver
san_ip = 192.168.2.192
san_login = manage
san_password =!manage
volume_backend_name=HPMSA_FC


[lvmdriver-1]
volume_group=stack-volumes-1
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=LVM_iSCSI



*I am getting below cinder.conf file after running stack.sh script*

[keystone_authtoken]
auth_uri = http://192.168.2.151:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = vedams123
admin_user = cinder
admin_tenant_name = service
cafile =
identity_uri = http://192.168.2.151:35357

[DEFAULT]
rabbit_password = vedams123
rabbit_hosts = 192.168.2.151
rpc_backend = cinder.openstack.common.rpc.impl_kombu
use_syslog = True
*default_volume_type = lvmdriver-1*
*enabled_backends = lvmdriver-1*
enable_v1_api = true
periodic_interval = 60
lock_path = /opt/stack/data/cinder
state_path = /opt/stack/data/cinder
osapi_volume_extension = cinder.api.contrib.standard_extensions
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_config = /etc/cinder/api-paste.ini
sql_connection = mysql://root:vedams123@127.0.0.1/cinder?charset=utf8
iscsi_helper = tgtadm
my_ip = 192.168.2.151
verbose = True
debug = True
auth_strategy = keystone

[lvmdriver-1]
volume_group = stack-volumes-1
volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name = LVM_iSCSI

[hp_msa_driver]
volume_backend_name = HPMSA_FC
san_password = !manage
san_login = manage
san_ip = 192.168.2.192
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver



*Then i analyzed source code of stack.sh,and added in local.conf this line:*
*CINDER_ENABLED_BACKENDS=hp_msa:hp_msa_driver,lvm:lvmdriver-1*


Now i am getting hp_msa_fc in cinder.conf in enabled_backends



Regards
Nikesh
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Doug Wiegley]

Hi all,

> Validations such as ³timeout > delay² should be performed on the API
>level before it reaches the driver.
For a configuration tree (lb, listeners, pools, etc.), there should be one
provider.

You¹re right, but I think the point of Vijay¹s example was to highlight
the combo error problem with populating all of the driver objects at once
(in short, the driver interface isn¹t well suited to that model.)  That
his one example can be covered by API validators is irrelevant.  Consider
a backend that does not support APP_COOKIE¹s, or HTTPS_TERMINATED (but has
multiple listeners) instead.  Should the entire load balancer create fail,
or should it offer degraded service?  Do all drivers have to implement a
transaction rollback; wait, the interface makes that very hard.  That¹s
his point.  The driver is no longer just glue code between interfaces;
it¹s now a mini-object error handler.


> Having provider defined in multiple places does not make sense.

Channeling Brandon, who can yell if I get this wrong, the point is not to
have a potentially different provider on each object.  It¹s to allow a
provider to be assigned when the first object in the tree is created, so
that future related objects will always get routed to the same provider.
Not knowing which provider should get all the objects is why we have to
wait until we see a LoadBalancer object.


All of this sort of edge case nonsense is because we (the royal we, the
community), wanted all load balancer objects to be ³root² objects, even
though only one of them is an actual root today, to support many-to-many
relationships among all of them, at some future date, without an interface
change.  If my bias is showing that I¹m not a fan of adding this
complexity for that, I¹m not surprised.

Thanks,
doug


On 8/11/14, 7:57 AM, "Samuel Bercovici"  wrote:

>Hi,
> 
>Validations such as ³timeout > delay² should be performed on the API
>level before it reaches the driver.
> 
>For a configuration tree (lb, listeners, pools, etc.), there should be
>one provider.
>
>Having provider defined in multiple places does not make sense.
> 
> 
>-San.
> 
> 
>From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]
>
>Sent: Monday, August 11, 2014 2:43 PM
>To: OpenStack Development Mailing List (openstack-dev@lists.openstack.org)
>Subject: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver
>interface on every API request"
>
>
> 
>Hi:
> 
>Continuing from last week¹s LBaaS meetingŠ
> 
>Currently an entity cannot be sent to driver unless it is linked to
>loadbalancer because loadbalancer is the root object and driver
>information is only available with loadbalancer.
>
> 
>The request to the driver is delayed because of which error propagation
>becomes tricky.
> 
>Let¹s say a monitor was configured with timeout > delay there would be no
>error then.
>When a listener is configured there will be a monitor creation/deployment
>error like ³timeout configured greater than delay².
> 
>Unless the error is very clearly crafted the user won¹t be able to
>understand the error.
> 
>I am half-heartedly OK with current approach.
>
> 
>But, I would prefer Brandon¹s Solution ­ make provider an attribute in
>each of the entities to get rid of this problem.
>
> 
>What do others think?
> 
>Thanks,
>Vijay V.
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Which program for Rally

[David Kranz]

On 08/06/2014 05:48 PM, John Griffith wrote:
I have to agree with Duncan here.  I also don't know if I fully 
understand the limit in options.  Stress test seems like it 
could/should be different (again overlap isn't a horrible thing) and I 
don't see it as siphoning off resources so not sure of the issue. 
 We've become quite wrapped up in projects, programs and the like 
lately and it seems to hinder forward progress more than anything else.


I'm also not convinced that Tempest is where all things belong, in 
fact I've been thinking more and more that a good bit of what Tempest 
does today should fall more on the responsibility of the projects 
themselves.  For example functional testing of features etc, ideally 
I'd love to have more of that fall on the projects and their 
respective teams.  That might even be something as simple to start as 
saying "if you contribute a new feature, you have to also provide a 
link to a contribution to the Tempest test-suite that checks it". 
 Sort of like we do for unit tests, cross-project tracking is 
difficult of course, but it's a start.  The other idea is maybe 
functional test harnesses live in their respective projects.


Honestly I think who better to write tests for a project than the 
folks building and contributing to the project.  At some point IMO the 
QA team isn't going to scale.  I wonder if maybe we should be thinking 
about proposals for delineating responsibility and goals in terms of 
functional testing?




All good points. Your last paragraph was discussed by the QA team 
leading up to and at the Atlanta summit. The conclusion was that the 
api/functional tests focused on a single project should be part of that 
project. As Sean said, we can envision there being half (or some other 
much smaller number) as many such tests in tempest going forward.


Details are under discussion, but the way this is likely to play out is 
that individual projects will start by creating their own functional 
tests outside of tempest. Swift already does this and neutron seems to 
be moving in that direction. There is a spec to break out parts of 
tempest 
(https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst) 
into a library that might be used by projects implementing functional 
tests.


Once a project has "sufficient" functional testing, we can consider 
removing its api tests from tempest. This is a bit tricky because 
tempest needs to cover *all* cross-project interactions. In this 
respect, there is no clear line in tempest between scenario tests which 
have this goal explicitly, and api tests which may also involve 
interactions that might not be covered in a scenario. So we will need a 
principled way to make sure there is complete cross-project coverage in 
tempest with a smaller number of api tests.


 -David
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Fuel] 6.0 blueprints cleanup

[Mike Scherbakov]

+2, yes please


On Mon, Aug 11, 2014 at 7:42 PM, Dmitry Borodaenko  wrote:

> All,
>
> Please refer to this email for a list of information that has to be
> populated in a blueprint before it can be assigned and scheduled:
> http://lists.openstack.org/pipermail/openstack-dev/2014-August/042042.html
>
> Please don't schedule a blueprint to 6.0 until it has these details
> and its assignees are confirmed.
>
> On Mon, Aug 11, 2014 at 8:30 AM, Dmitry Pyzhov 
> wrote:
> > We've moved all blueprints from 6.0 to 'next' milestone. It has been
> done in
> > order to better view of stuff that we really want to implement in 6.0.
> >
> > Feature freeze for 6.0 release is planned to 18th of September. If you
> are
> > going to merge your blueprint before that date, you can move it to 6.0
> > milestone and 6.0.x series. But blueprint must have fixed scope and must
> be
> > assigned to person who will lead this activity.
> >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
>
> --
> Dmitry Borodaenko
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Mike Scherbakov
#mihgen
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

[Gary Kotton]

On 8/11/14, 6:06 PM, "Dan Smith"  wrote:

>> As the person who -2'd the review, I'm thankful you raised this issue on
>> the ML, Jay. Much appreciated.
>
>The "metadetails" term isn't being invented in this patch, of course. I
>originally complained about the difference when this was being added:
>
>https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contr
>ib/server_groups.py,cm
>
>As best I can tell, the response in that patch set about why it's being
>translated is wrong (backwards). I expect that the API extension at the
>time called it "metadetails" and they decided to make the object the
>same and do the translation there.
>
>From what I can tell, the actual server_group API extension that made it
>into the tree never got the ability to set/change/etc the
>metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
>wrongly.
>
>If we care to have this functionality, then I propose we change the
>attribute on the object (we can handle this with versioning) and reflect
>it as "metadata" in the API.
>
>However, I have to ask: do we really need another distinct metadata
>store attached to server_groups? If not, how about we just remove it
>from the database and the object, clean up the bit of residue that is
>still in the API extension and be done with it?

The initial version of the feature did not make use of this. The reason
was that we chose for a very
Limited subset to be used, that is, affinity and anti affinity. Moving
forwards we would like to implement
A number of different policies with this. We can drop it at the moment due
to the fact that it is not used.

I think that Yathi may be using this for the constrain scheduler. But I am
not 100% sure.

>
>--Dan
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Use cases with regards to VIP and routers

[Doug Wiegley]

Hi Susanne,

While there are a few operators involved with LBaaS that would have good
input, you might want to also ask this on the non-dev mailing list, for a
larger sample size.

Thanks,
doug

On 8/11/14, 3:05 AM, "Susanne Balle"  wrote:

>Gang,
>I was asked the following questions around our Neutron LBaaS use cases:
>1.  Will there be a scenario where the ³VIP² port will be in a different
>Node, from all the Member ³VMs² in a pool.
>
>
>2.  Also how likely is it for the LBaaS configured subnet to not have a
>³router² and just use the ³extra_routes²
> option.
>3.  Is there a valid use case where customers will be using the
>³extra_routes² with subnets instead of the ³routers².
> ( It would be great if you have some use case picture for this).
>Feel free to chime in here and I'll summaries the answers.
>Regards Susanne
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] [third-party] Freescale CI log site is being blocked

[trinath.soman...@freescale.com]

Hi-

Today I contacted the service provider regarding the malware in the website. 
Got a response that website is truly functional and malware free.

I too have verified all the directories, subdirectories and log files manually 
for any malware injected into the website. I have detected none.

There is no anonymous login enabled for FTP or cPanel login to the server. The 
FTP is protected with Strong Passcode.

This is an update from my end. 

Like all other CI's now normal browsing of logs is available for view.

The CI is made down to take the above explained changes to take place. Now the 
CI is active running the jobs.

All the old logs are present as in place. 

You may browse the old logs using the url 

For ML2 Mechanism driver : 
http://fslopenstackci.com/{change_number}/{change_patchset}/Freescale-ML2-Mechanism-Driver
For FWaaS Plugin : 
http://fslopenstackci.com/{change_number}/{change_patchset}/Freescale-FWaaS-Plugin

Now I have updated the CI to create a BUILD directory as well to showcase logs 
for "rechecks".

With this new change the log URL will be 

For ML2 Mechanism driver : 
http://fslopenstackci.com/{build_number}/{change_number}/{change_patchset}/Freescale-ML2-Mechanism-Driver
For FWaaS Plugin : 
http://fslopenstackci.com/{build_number}/{change_number}/{change_patchset}/Freescale-FWaaS-Plugin


Hi Mestery-

Kindly please verify the access to the site. 

Also, if it's still in blocking mode, kindly mail me the logs with Cisco WSA to 
verify the reason behind this blocking.

Kindly help me with your review on my code at 
https://review.openstack.org/#/c/109659/

Thanking  you all.

--
Trinath Somanchi - B39208
trinath.soman...@freescale.com | Mob: +91 9866 235 130

-Original Message-
From: Anita Kuno [mailto:ante...@anteaya.info] 
Sent: Saturday, August 09, 2014 11:09 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [neutron] [third-party] Freescale CI log site is 
being blocked

On 08/08/2014 11:27 PM, trinath.soman...@freescale.com wrote:
> Thanks anita for the reply.
> 
> Previously the existing server is accessible by kyle. But now its not being 
> accessible. 
> 
> For the paid hosting I have its administered by godaddy
If you are paying godaddy to administer the server, have you asked them why one 
of your users has acknowledged your site is blacklisted by Cisco WSA appliances?

If you are paying them to administer your server, answering your question falls 
within their job.

You need to find out the reason behind Cisco security blocking, that is what I 
am asking you to do. It if fine if you don't know, but it is your 
responsibility to find out.

Thanks Trinath,
Anita.

> and the FTP is only accessed by Jenkins. 
> 
> I can try relocating FTP web based file browser script and provide a normal 
> view of files. 
> 
> Don't know the reason behind Cisco Security blocking the access where it has 
> given access to view the website before.
> 
> Thanks a lot again for the brief email.
> 
> 
> --
> Trinath Somanchi - B39208
> trinath.soman...@freescale.com | extn: 4048
> 
> -Original Message-
> From: Anita Kuno [mailto:ante...@anteaya.info]
> Sent: Saturday, August 09, 2014 10:21 AM
> To: openstack-dev@lists.openstack.org
> Subject: Re: [openstack-dev] [neutron] [third-party] Freescale CI log 
> site is being blocked
> 
> On 08/08/2014 10:06 PM, trinath.soman...@freescale.com wrote:
>> Hi Sumit-
>>
>> When I try to paste a large log text into paste.openstack, It is giving me 
>> image verification and says its spam.
> Let's not confuse paste.openstack.org's spam blocker from spam blockers on 
> servers. They are two separate functionalities and the conversation does not 
> move forward if we try to pretend they are the same thing or even remotely 
> related, which they are not.
> 
> If you recall, Trinath, the first server you had got hacked since you had not 
> hardened it appropriately. Having hosting via go daddy or any other paid 
> hosting service does not absolve you of the responsibility of having a well 
> maintained server. If you need help maintaining your server, I suggest you 
> contract a server administrator to advise you or do the work. We have to 
> assume a certain level of competence here, due to the responsibility involved 
> I don't think you are going to get many responses to questions if you don't 
> know how to maintain your server.
> This isn't really the the place to ask. Running your third party ci system 
> and copying the logs, sure this is the place, basic server maintenance is 
> your responsibility.
> 
> If you recall, a basic evaluation of your server logs told you you had been 
> hacked the last time. This might be a place to start now.
> 
> In any case, please maintain your server and please address Kyle's concerns.
> 
> Thank you Trianth,
> Anita.
>>
>> I don't know why its taken as spam/malware. It's a paid hosting I had from 
>> GODADDY.
>>
>> --
>> Trinath Somanchi - B39208
>> trinath.soman...@freescale.com | e

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Samuel Bercovici]

Hi Doug,

In some implementations Driver !== Device. I think this is also true for HA 
Proxy.
This might mean that there is a difference between creating a logical object 
and when there is enough information to actually schedule/place this into a 
device.
The ability to express such errors (detecting an error on a logical object 
after it was created but when it actually get scheduled) should be discussed 
and addressed anyway.

-Sam.


-Original Message-
From: Doug Wiegley [mailto:do...@a10networks.com] 
Sent: Monday, August 11, 2014 6:55 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver 
interface on every API request"

Hi all,

> Validations such as ³timeout > delay² should be performed on the API 
>level before it reaches the driver.
For a configuration tree (lb, listeners, pools, etc.), there should be one 
provider.

You¹re right, but I think the point of Vijay¹s example was to highlight the 
combo error problem with populating all of the driver objects at once (in 
short, the driver interface isn¹t well suited to that model.)  That his one 
example can be covered by API validators is irrelevant.  Consider a backend 
that does not support APP_COOKIE¹s, or HTTPS_TERMINATED (but has multiple 
listeners) instead.  Should the entire load balancer create fail, or should it 
offer degraded service?  Do all drivers have to implement a transaction 
rollback; wait, the interface makes that very hard.  That¹s his point.  The 
driver is no longer just glue code between interfaces; it¹s now a mini-object 
error handler.


> Having provider defined in multiple places does not make sense.

Channeling Brandon, who can yell if I get this wrong, the point is not to have 
a potentially different provider on each object.  It¹s to allow a provider to 
be assigned when the first object in the tree is created, so that future 
related objects will always get routed to the same provider.
Not knowing which provider should get all the objects is why we have to wait 
until we see a LoadBalancer object.


All of this sort of edge case nonsense is because we (the royal we, the 
community), wanted all load balancer objects to be ³root² objects, even though 
only one of them is an actual root today, to support many-to-many relationships 
among all of them, at some future date, without an interface change.  If my 
bias is showing that I¹m not a fan of adding this complexity for that, I¹m not 
surprised.

Thanks,
doug


On 8/11/14, 7:57 AM, "Samuel Bercovici"  wrote:

>Hi,
> 
>Validations such as ³timeout > delay² should be performed on the API 
>level before it reaches the driver.
> 
>For a configuration tree (lb, listeners, pools, etc.), there should be 
>one provider.
>
>Having provider defined in multiple places does not make sense.
> 
> 
>-San.
> 
> 
>From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]
>
>Sent: Monday, August 11, 2014 2:43 PM
>To: OpenStack Development Mailing List 
>(openstack-dev@lists.openstack.org)
>Subject: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver 
>interface on every API request"
>
>
> 
>Hi:
> 
>Continuing from last week¹s LBaaS meetingŠ
> 
>Currently an entity cannot be sent to driver unless it is linked to 
>loadbalancer because loadbalancer is the root object and driver 
>information is only available with loadbalancer.
>
> 
>The request to the driver is delayed because of which error propagation 
>becomes tricky.
> 
>Let¹s say a monitor was configured with timeout > delay there would be 
>no error then.
>When a listener is configured there will be a monitor 
>creation/deployment error like ³timeout configured greater than delay².
> 
>Unless the error is very clearly crafted the user won¹t be able to 
>understand the error.
> 
>I am half-heartedly OK with current approach.
>
> 
>But, I would prefer Brandon¹s Solution ­ make provider an attribute in 
>each of the entities to get rid of this problem.
>
> 
>What do others think?
> 
>Thanks,
>Vijay V.
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] devstack local.conf file

[Asselin, Ramy]

Hi Nikesh,

You need to set the enabled_backends in the local.conf file.
e.g.

[[post-config|$CINDER_CONF]]
[DEFAULT]
enabled_backends=hp_msa_driver
[hp_msa_driver]
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver
…

Ramy

From: Nikesh Kumar Mahalka [mailto:nikeshmaha...@vedams.com]
Sent: Monday, August 11, 2014 8:55 AM
To: openst...@lists.openstack.org; openstack-dev@lists.openstack.org
Subject: [openstack-dev] devstack local.conf file

Hi,
I have gone through devstack links.
They are not clear like openstack.org documents.


For Example:
when i am using below local.conf file in devstack,"hp_msa_driver" is not coming 
in "enabled_backends" in cinder.conf after running stack.sh.

[[local|localrc]]
ADMIN_PASSWORD=vedams123
DATABASE_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
SERVICE_TOKEN=ADMIN
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.2.170/29
HOST_IP=192.168.2.151
LOGFILE=$DEST/logs/stack.sh.log
SCREEN_LOGDIR=$DEST/logs/screen
SYSLOG=True
SYSLOG_HOST=$HOST_IP
SYSLOG_PORT=516
TEMPEST_VOLUME_DRIVER=hp_msa_fc
TEMPEST_VOLUME_VENDOR="Hewlett-Packard"
TEMPEST_STORAGE_PROTOCOL=FC


[[post-config|$CINDER_CONF]]
[hp_msa_driver]
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver
san_ip = 192.168.2.192
san_login = manage
san_password =!manage
volume_backend_name=HPMSA_FC


[lvmdriver-1]
volume_group=stack-volumes-1
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name=LVM_iSCSI



I am getting below cinder.conf file after running stack.sh script

[keystone_authtoken]
auth_uri = http://192.168.2.151:5000/v2.0
signing_dir = /var/cache/cinder
admin_password = vedams123
admin_user = cinder
admin_tenant_name = service
cafile =
identity_uri = http://192.168.2.151:35357

[DEFAULT]
rabbit_password = vedams123
rabbit_hosts = 192.168.2.151
rpc_backend = cinder.openstack.common.rpc.impl_kombu
use_syslog = True
default_volume_type = lvmdriver-1
enabled_backends = lvmdriver-1
enable_v1_api = true
periodic_interval = 60
lock_path = /opt/stack/data/cinder
state_path = /opt/stack/data/cinder
osapi_volume_extension = cinder.api.contrib.standard_extensions
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_config = /etc/cinder/api-paste.ini
sql_connection = 
mysql://root:vedams123@127.0.0.1/cinder?charset=utf8
iscsi_helper = tgtadm
my_ip = 192.168.2.151
verbose = True
debug = True
auth_strategy = keystone

[lvmdriver-1]
volume_group = stack-volumes-1
volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver
volume_backend_name = LVM_iSCSI

[hp_msa_driver]
volume_backend_name = HPMSA_FC
san_password = !manage
san_login = manage
san_ip = 192.168.2.192
volume_driver = cinder.volume.drivers.san.hp.hp_msa_fc.HPMSAFCDriver



Then i analyzed source code of stack.sh,and added in local.conf this line:
CINDER_ENABLED_BACKENDS=hp_msa:hp_msa_driver,lvm:lvmdriver-1


Now i am getting hp_msa_fc in cinder.conf in enabled_backends



Regards
Nikesh
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Doug Wiegley]

Hi Sam,

Very true.  I think that Vijay’s objection is that we are currently
imposing a logical structure on the driver, when it should be a driver
decision.  Certainly, it goes both ways.

And I also agree that the mechanism for returning multiple errors, and the
ability to specify whether those errors are fatal or not, individually, is
currently weak.

Doug


On 8/11/14, 10:21 AM, "Samuel Bercovici"  wrote:

>Hi Doug,
>
>In some implementations Driver !== Device. I think this is also true for
>HA Proxy.
>This might mean that there is a difference between creating a logical
>object and when there is enough information to actually schedule/place
>this into a device.
>The ability to express such errors (detecting an error on a logical
>object after it was created but when it actually get scheduled) should be
>discussed and addressed anyway.
>
>-Sam.
>
>
>-Original Message-
>From: Doug Wiegley [mailto:do...@a10networks.com]
>Sent: Monday, August 11, 2014 6:55 PM
>To: OpenStack Development Mailing List (not for usage questions)
>Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling
>driver interface on every API request"
>
>Hi all,
>
>> Validations such as ³timeout > delay² should be performed on the API
>>level before it reaches the driver.
>For a configuration tree (lb, listeners, pools, etc.), there should be
>one provider.
>
>You¹re right, but I think the point of Vijay¹s example was to highlight
>the combo error problem with populating all of the driver objects at once
>(in short, the driver interface isn¹t well suited to that model.)  That
>his one example can be covered by API validators is irrelevant.  Consider
>a backend that does not support APP_COOKIE¹s, or HTTPS_TERMINATED (but
>has multiple listeners) instead.  Should the entire load balancer create
>fail, or should it offer degraded service?  Do all drivers have to
>implement a transaction rollback; wait, the interface makes that very
>hard.  That¹s his point.  The driver is no longer just glue code between
>interfaces; it¹s now a mini-object error handler.
>
>
>> Having provider defined in multiple places does not make sense.
>
>Channeling Brandon, who can yell if I get this wrong, the point is not to
>have a potentially different provider on each object.  It¹s to allow a
>provider to be assigned when the first object in the tree is created, so
>that future related objects will always get routed to the same provider.
>Not knowing which provider should get all the objects is why we have to
>wait until we see a LoadBalancer object.
>
>
>All of this sort of edge case nonsense is because we (the royal we, the
>community), wanted all load balancer objects to be ³root² objects, even
>though only one of them is an actual root today, to support many-to-many
>relationships among all of them, at some future date, without an
>interface change.  If my bias is showing that I¹m not a fan of adding
>this complexity for that, I¹m not surprised.
>
>Thanks,
>doug
>
>
>On 8/11/14, 7:57 AM, "Samuel Bercovici"  wrote:
>
>>Hi,
>> 
>>Validations such as ³timeout > delay² should be performed on the API
>>level before it reaches the driver.
>> 
>>For a configuration tree (lb, listeners, pools, etc.), there should be
>>one provider.
>>
>>Having provider defined in multiple places does not make sense.
>> 
>> 
>>-San.
>> 
>> 
>>From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]
>>
>>Sent: Monday, August 11, 2014 2:43 PM
>>To: OpenStack Development Mailing List
>>(openstack-dev@lists.openstack.org)
>>Subject: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver
>>interface on every API request"
>>
>>
>> 
>>Hi:
>> 
>>Continuing from last week¹s LBaaS meetingŠ
>> 
>>Currently an entity cannot be sent to driver unless it is linked to
>>loadbalancer because loadbalancer is the root object and driver
>>information is only available with loadbalancer.
>>
>> 
>>The request to the driver is delayed because of which error propagation
>>becomes tricky.
>> 
>>Let¹s say a monitor was configured with timeout > delay there would be
>>no error then.
>>When a listener is configured there will be a monitor
>>creation/deployment error like ³timeout configured greater than delay².
>> 
>>Unless the error is very clearly crafted the user won¹t be able to
>>understand the error.
>> 
>>I am half-heartedly OK with current approach.
>>
>> 
>>But, I would prefer Brandon¹s Solution ­ make provider an attribute in
>>each of the entities to get rid of this problem.
>>
>> 
>>What do others think?
>> 
>>Thanks,
>>Vijay V.
>>
>
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list

Re: [openstack-dev] [OpenStack][InstanceGroup] metadetails and metadata in instance_group.py

[Sylvain Bauza]

Le 11/08/2014 18:03, Gary Kotton a écrit :


On 8/11/14, 6:06 PM, "Dan Smith"  wrote:


As the person who -2'd the review, I'm thankful you raised this issue on
the ML, Jay. Much appreciated.

The "metadetails" term isn't being invented in this patch, of course. I
originally complained about the difference when this was being added:

https://review.openstack.org/#/c/109505/1/nova/api/openstack/compute/contr
ib/server_groups.py,cm

As best I can tell, the response in that patch set about why it's being
translated is wrong (backwards). I expect that the API extension at the
time called it "metadetails" and they decided to make the object the
same and do the translation there.


>From what I can tell, the actual server_group API extension that made it

into the tree never got the ability to set/change/etc the
metadata/metadetails anyway, so there's no reason (AFAICT) to add it in
wrongly.

If we care to have this functionality, then I propose we change the
attribute on the object (we can handle this with versioning) and reflect
it as "metadata" in the API.

However, I have to ask: do we really need another distinct metadata
store attached to server_groups? If not, how about we just remove it

>from the database and the object, clean up the bit of residue that is

still in the API extension and be done with it?

The initial version of the feature did not make use of this. The reason
was that we chose for a very
Limited subset to be used, that is, affinity and anti affinity. Moving
forwards we would like to implement
A number of different policies with this. We can drop it at the moment due
to the fact that it is not used.

I think that Yathi may be using this for the constrain scheduler. But I am
not 100% sure.



Unless I'm wrong, I can't see where this metadata is being used in the 
scheduler, either for filtering or for other reasons.


So, please give us context why this is currently useful ?

If this is something for the next future, I would love discussing it 
with regards to the current split.



Thanks,
-Sylvain


--Dan



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree

[Robert Kukura]

On 8/8/14, 6:28 PM, Salvatore Orlando wrote:
"If we want to keep everything the way it is, we have to change 
everything" [1]


This is pretty much how I felt after reading this proposal, and I felt 
that this quote, which Ivar will probably appreciate, was apt to the 
situation.
Recent events have spurred a discussion about the need for a change in 
process. It is not uncommon indeed to believe that by fixing the 
process, things will inevitably change for better. While no-one argues 
that flaws in processes need to be fixed, no process change will ever 
change anything, in my opinion, unless it is aimed at spurring change 
in people as well.


From what I understand, this proposal starts with the assumption that 
any new feature which is committed to Neutron (ie: has a blueprint 
approved), and is not a required neutron component should be 
considered as a preview. This is not different from the process, 
which, albeit more informally, has been adopted so far. Load 
Balancing, Firewall, VPN, have all been explicitly documented as 
experimental in their first release; I would argue that even if not 
experimental anymore, they may not be considered stable until their 
stability was proven by upstream QA with API and scenario tests - but 
this is not sanctioned anywhere currently, I think.
Correct, this proposal is not so much a new process or change in process 
as a formalization of what we've already been doing, and a suggested 
adaptation to clarify the current expectations around stability of new APIs.


According to this proposal, for preview features:
- all the code would be moved to a "preview" package

Yes.

- Options will be marked as "preview"

Yes.

- URIs should be prefixed with "preview"
That's what I suggested, but, as several people have pointed out, this 
does seem like its worth the cost of breaking the API compatibility just 
at the point when it is being declared stable. I'd like to withdraw this 
item.

- CLIs will note the features are "preview" in their help strings

Yes.
- Documentation will explicitly state this feature is "preview" (I 
think we already mark them as experimental, frankly I don't think 
there are a lot of differences in terminology here)
Yes. Again to me, failure is one likely outcome of an "experiment". The 
term "preview" is intended to imply more of a commitment to quickly 
reach stability.

- Database migrations will be in the main alembic path as usual

Right.

- CLI, Devstack and Heat support will be available

Right, as appropriate for the feature.

- Can be used by non-preview neutron code
No, I suggested "No non-preview Neutron code should import code from 
anywhere under the neutron.preview module, ...".

- Will undergo the usual review process
Right. This is key for the code to not have to jump through a new major 
upheaval at right as it becomes stable.
- QA will be desirable, but will done either with "WIP" tempest 
patches or merging the relevant scenario tests in the preview feature 
iself
More than "desirable". We need a way to maintain and run the 
tempest-like API and scenario tests during the stabilization process, 
but to let then evolve with the feature.
- The feature might be promoted or removed, but the process for this 
is not yet defined.
Any suggestions? I did try to address preventing long-term stagnation of 
preview features. As a starting point, reviewing and merging a patch 
that moves the code from the preview sub-tree to its intended location 
could be a lightweight promotion process.


I don't think this change in process will actually encourage better 
behaviour both by contributors and core reviewers.
Encouraging better behavior might be necessary, but wasn't the main 
intent of this proposal. This proposal was intended to clarify and 
formalize the stability expectations around the initial releases of new 
features. It was specifically intended to address the conundrum 
currently faced by reviewers regarding patches that meet all applicable 
quality standards, but may not yet have (somehow, miraculously) achieved 
the maturity associated with stable APIs and features fully supported 
for widespread deployment.
I reckon that better behaviour might be encouraged by forcing 
developer and reviewers to merge in the neutron source code tree only 
code which meets the highest quality standards. A change in process 
should enforce this - and when I think about the criteria, I think at 
the same kind of criteria we're being imposed to declare parity with 
nova. Proven reliability, and scalability should be a must. Proven 
usability should be a requirement for all new APIs.
I agree regarding the quality standards for merging of code, and am not 
suggesting relaxing those one bit. But proving all of the desirable 
system-level attributes of a complex new feature before merging anything 
precludes any kind of iterative development process. I think we should 
consider enforcing things like proven reliability, scalability, and 
usability at th

Re: [openstack-dev] [designate] [neutron] designate and neutron integration

[Carl Baldwin]

kazuhiro MIYASHITA,

I have done a lot of thinking about this.  I have a blueprint on hold
until Kilo for Neutron/Designate integration [1].

However, my blueprint doesn't quite address what you are going after
here.  An assumption that I have made is that Designate is an external
or internet facing service so a Neutron router needs to be in the
datapath to carry requests from dnsmasq to an external network.  The
advantage of this is that it is how Neutron works today so there is no
new development needed.

Could you elaborate on the advantages of connecting dnsmasq directly
to the external network where Designate will be available?

Carl

[1] https://review.openstack.org/#/c/88624/

On Mon, Aug 11, 2014 at 7:51 AM, Miyashita, Kazuhiro
 wrote:
> Hi,
>
> I want to ask about neutron and designate integration.
> I think dnsmasq fowards DNS request from instance to designate is better.
>
>++
>|DNS server(designate)   |
>++
> |
> -+--+-- Network1
>  |
>   ++
>   |dnsmasq |
>   ++
> |
> -+--+-- Network2
>  |
> +-+
> |instance |
> +-+
>
> Because it's simpler than virtual router connects Network1 and Network2.
> If router connects network, instance should know where DNS server is. it is 
> complicated.
> dnsmasq returns its ip address as dns server in DHCP replay by ordinary, so,
> I think good dnsmasq becomes like a gateway to designate.
>
> But, I can't connect dnsmasq to Network1. because of today's neutron design.
>
> Question:
>   Does designate design team have a plan such as above integration?
>   or other integration design?
>
> *1: Network1 and Network2 are deployed by neutron.
> *2: neutron deploys dnsmasq as a dhcp server.
> dnsmasq can forward DNS request.
>
> Thanks,
>
> kazuhiro MIYASHITA
>
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: FW: [Neutron] Group Based Policy and the way forward

[Hemanth Ravi]

On Fri, Aug 8, 2014 at 7:13 PM, Armando M.  wrote:

>
>> On Fri, Aug 8, 2014 at 5:38 PM, Armando M.  wrote:
>>
>>>
>>>
   One advantage of the service plugin is that one can leverage the
 neutron common framework such as Keystone authentication where common
 scoping is done. It would be important in the policy type of framework to
 have such scoping

>>>
>>> The framework you're referring to is common and already reusable, it's
>>> not a prerogative of Neutron.
>>>
>>
>>  Are you suggesting that Service Plugins, L3, IPAM etc become individual
>> endpoints, resulting in redundant authentication round-trips for each of
>> the components.
>>
>> Wouldn't this result in degraded performance and potential consistency
>> issues?
>>
>
> The endpoint - in the OpenStack lingo - that exposes the API abstractions
> (concepts and operations) can be, logically and physically, different from
> the worker that implements these abstractions; authentication is orthogonal
> to this and I am not suggesting what you mention.
>

>From what I understand, you are saying that the implementation could be
done via a mechanism different than a service plugin. Would this be done by
implementing the service plugin as a different process? This would imply
making changes to the the neutron server - plugin interface. If this is the
case, wouldn't it be better to use the existing mechanism to avoid
introducing any instability at this stage of the Juno cycle.


> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [designate] [neutron] designate and neutron integration

[Hayes, Graham]

kazuhiro MIYASHITA,

As designate progresses with server pools, we aim to have support for a
'private' dns server, that could run within a neutron network - is that
the level of integration you were referring to?

That is, for the time being, a long term goal, and not covered by Carl's
Kilo blueprint.

We talked with both people from both Neutron and Nova in Atlanta, and
worked out the first steps for designate / neutron integration (auto
provisioning of records)

For that level of integration, we are assuming that a neutron router
will be involved in DNS queries within a network.

Long term I would prefer to see a 'private pool' connecting directly to
the Network2 (like any other service VM (LBaaS etc)) and have dnsmasq
pass on only records hosted by that 'private pool' to designate.

This is all yet to be fleshed out, so I am open to suggestions. It
requires that we complete server pools, and that work is only just
starting (it was the main focus of our mid-cycle 2 weeks ago).

Graham

On Mon, 2014-08-11 at 11:02 -0600, Carl Baldwin wrote:
> kazuhiro MIYASHITA,
> 
> I have done a lot of thinking about this.  I have a blueprint on hold
> until Kilo for Neutron/Designate integration [1].
> 
> However, my blueprint doesn't quite address what you are going after
> here.  An assumption that I have made is that Designate is an external
> or internet facing service so a Neutron router needs to be in the
> datapath to carry requests from dnsmasq to an external network.  The
> advantage of this is that it is how Neutron works today so there is no
> new development needed.
> 
> Could you elaborate on the advantages of connecting dnsmasq directly
> to the external network where Designate will be available?
> 
> Carl
> 
> [1] https://review.openstack.org/#/c/88624/
> 
> On Mon, Aug 11, 2014 at 7:51 AM, Miyashita, Kazuhiro
>  wrote:
> > Hi,
> >
> > I want to ask about neutron and designate integration.
> > I think dnsmasq fowards DNS request from instance to designate is better.
> >
> >++
> >|DNS server(designate)   |
> >++
> > |
> > -+--+-- Network1
> >  |
> >   ++
> >   |dnsmasq |
> >   ++
> > |
> > -+--+-- Network2
> >  |
> > +-+
> > |instance |
> > +-+
> >
> > Because it's simpler than virtual router connects Network1 and Network2.
> > If router connects network, instance should know where DNS server is. it is 
> > complicated.
> > dnsmasq returns its ip address as dns server in DHCP replay by ordinary, so,
> > I think good dnsmasq becomes like a gateway to designate.
> >
> > But, I can't connect dnsmasq to Network1. because of today's neutron design.
> >
> > Question:
> >   Does designate design team have a plan such as above integration?
> >   or other integration design?
> >
> > *1: Network1 and Network2 are deployed by neutron.
> > *2: neutron deploys dnsmasq as a dhcp server.
> > dnsmasq can forward DNS request.
> >
> > Thanks,
> >
> > kazuhiro MIYASHITA
> >
> >
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] oslo.concurrency repo review

[Yuriy Taraday]

On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow  wrote:

> One question from me:
>
> Will there be later fixes to remove oslo.config dependency/usage from
> oslo.concurrency?
>
> I still don't understand how oslo.concurrency can be used as a library
> with the configuration being set in a static manner via oslo.config (let's
> use the example of `lock_path` @ https://github.com/YorikSar/
> oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). For
> example:
>
> Library X inside application Z uses lockutils (via the nice
> oslo.concurrency library) and sets the configuration `lock_path` to its
> desired settings, then library Y (also a user of oslo.concurrency) inside
> same application Z sets the configuration for `lock_path` to its desired
> settings. Now both have some unknown set of configuration they have set and
> when library X (or Y) continues to use lockutils they will be using some
> mix of configuration (likely some mish mash of settings set by X and Y);
> perhaps to a `lock_path` that neither actually wants to be able to write
> to...
>
> This doesn't seem like it will end well; and will just cause headaches
> during debug sessions, testing, integration and more...
>
> The same question can be asked about the `set_defaults()` function, how is
> library Y or X expected to use this (are they?)??
>
> I hope one of the later changes is to remove/fix this??
>
> Thoughts?
>
> -Josh


I'd be happy to remove lock_path config variable altogether. It's basically
never used. There are two basic branches in code wrt lock_path:
- when you provide lock_path argument to lock (and derivative functions),
file-based lock is used and CONF.lock_path is ignored;
- when you don't provide lock_path in arguments, semaphore-based lock is
used and CONF.lock_path is just a prefix for its name (before hashing).

I wonder if users even set lock_path in their configs as it has almost no
effect. So I'm all for removing it, but...
>From what I understand, every major change in lockutils drags along a lot
of headache for everybody (and risk of bugs that would be discovered very
late). So is such change really worth it? And if so, it will require very
thorough research of lockutils usage patterns.

-- 

Kind regards, Yuriy.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Which program for Rally

[Zane Bitter]

On 08/08/14 10:41, Anne Gentle wrote:

- Would have to ensure Rally is what we want "first" as getting to be PTL
since you are first to propose seems to be the model.


I know that at one time it was popular in the trade/gutter press to cast 
aspersions on new projects by saying that someone getting to be a PTL 
was the major motivation behind them. And although, having been there, I 
can tell you that this was grossly unfair to the people concerned, at 
least you could see where the impression might have come from in the 
days where being a PTL guaranteed you a seat on the TC.


These days with a directly elected TC, the job of a PTL is confined to 
administrative busywork. To the extent that a PTL holds any real ex 
officio power, which is not a great extent, it's probably a mistake that 
will soon be rectified. If anyone is really motivated to become a PTL by 
their dreams of avarice then I can guarantee that they will be disappointed.


It seems pretty clear to me that projects want their own programs 
because they don't think it wise to hand over control of all changes to 
the thing they've been working on for the past year or more to a group 
of people who have barely glanced at it before and already have other 
priorities. I submit that this is sufficient to completely explain the 
proliferation of programs without attributing to anyone any untoward 
motivations.


Finally, *yes*, the model is indeed that the project working in the open 
with the community eventually gets incubated, and the proprietary 
project working behind closed doors with a plan to "'open source' it one 
day, when it's perfect" is doomed to perpetual irrelevance. You'll note 
that anyone who is unhappy about that still has an obvious course of 
action that doesn't involve punishing the people who are trying to do 
the Right Thing by the community.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

[Brad Topol]

Hi Eoghan,

Thanks for the note below.  However, one thing the overview below does not 
 cover  is why InfluxDB (http://influxdb.com/) is not being leveraged. 
Many folks feel that this technology is a viable solution for the problem 
space discussed below.

Thanks,

Brad


Brad Topol, Ph.D.
IBM Distinguished Engineer
OpenStack
(919) 543-0646
Internet:  bto...@us.ibm.com
Assistant: Kendra Witherspoon (919) 254-0680



From:   Eoghan Glynn 
To: "OpenStack Development Mailing List (not for usage questions)" 
, 
Date:   08/06/2014 11:17 AM
Subject:[openstack-dev] [tc][ceilometer] Some background on the 
gnocchi project




Folks,

It's come to our attention that some key individuals are not
fully up-to-date on gnocchi activities, so it being a good and
healthy thing to ensure we're as communicative as possible about
our roadmap, I've provided a high-level overview here of our
thinking. This is intended as a precursor to further discussion
with the TC.

Cheers,
Eoghan


What gnocchi is:
===

Gnocchi is a separate, but related, project spun up on stackforge
by Julien Danjou, with the objective of providing efficient
storage and retrieval of timeseries-oriented data and resource
representations.

The goal is to experiment with a potential approach to addressing
an architectural misstep made in the very earliest days of
ceilometer, specifically the decision to store snapshots of some
resource metadata alongside each metric datapoint. The core idea
is to move to storing datapoints shorn of metadata, and instead
allow the resource-state timeline to be reconstructed more cheaply
from much less frequently occurring events (e.g. instance resizes
or migrations).


What gnocchi isn't:
==

Gnocchi is not a large-scale under-the-radar rewrite of a core
OpenStack component along the lines of keystone-lite.

The change is concentrated on the final data-storage phase of
the ceilometer pipeline, so will have little initial impact on the
data-acquiring agents, or on transformation phase.

We've been totally open at the Atlanta summit and other forums
about this approach being a multi-cycle effort.


Why we decided to do it this way:


The intent behind spinning up a separate project on stackforge
was to allow the work progress at arms-length from ceilometer,
allowing normalcy to be maintained on the core project and a
rapid rate of innovation on gnocchi.

Note that that the developers primarily contributing to gnocchi
represent a cross-section of the core team, and there's a regular
feedback loop in the form of a recurring agenda item at the
weekly team meeting to avoid the effort becoming silo'd.


But isn't re-architecting frowned upon?
==

Well, the architecture of other OpenStack projects have also
under-gone change as the community understanding of the
implications of prior design decisions has evolved.

Take for example the move towards nova no-db-compute & the
unified-object-model in order to address issues in the nova
architecture that made progress towards rolling upgrades
unneccessarily difficult.

The point, in my understanding, is not to avoid doing the
course-correction where it's deemed necessary. Rather, the
principle is more that these corrections happen in an open
and planned way.


The path forward:


A subset of the ceilometer community will continue to work on
gnocchi in parallel with the ceilometer core over the remainder
of the Juno cycle and into the Kilo timeframe. The goal is to
have an initial implementation of gnocchi ready for tech preview
by the end of Juno, and to have the integration/migration/
co-existence questions addressed in Kilo.

Moving the ceilometer core to using gnocchi will be contingent
on it demonstrating the required performance characteristics and
providing the semantics needed to support a v3 ceilometer API
that's fit-for-purpose.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

[Clint Byrum]

Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:
> On 11/08/14 10:46, Clint Byrum wrote:
> > Right now we're stuck with an update that just doesn't work. It isn't
> > just about update-failure-recovery, which is coming along nicely, but
> > it is also about the lack of signals to control rebuild, poor support
> > for addressing machines as groups, and unacceptable performance in
> > large stacks.
> 
> Are there blueprints/bugs filed for all of these issues?
> 

Convergnce addresses the poor performance for large stacks in general.
We also have this:

https://bugs.launchpad.net/heat/+bug/1306743

Which shows how slow metadata access can get. I have worked on patches
but haven't been able to complete them. We made big strides but we are
at a point where 40 nodes polling Heat every 30s is too much for one CPU
to handle. When we scaled Heat out onto more CPUs on one box by forking
we ran into eventlet issues. We also ran into issues because even with
many processes we can only use one to resolve templates for a single
stack during update, which was also excessively slow.

We haven't been able to come back around to those yet, but you can see
where this has turned into a bit of a rat hole of optimization.

action-aware-sw-config is sort of what we want for rebuild. We
collaborated with the trove devs on how to also address it for resize
a while back but I have lost track of that work as it has taken a back
seat to more pressing issues.

Addressing groups is a general problem that I've had a hard time
articulating in the past. Tomas Sedovic has done a good job with this
TripleO spec, but I don't know that we've asked for an explicit change
in a bug or spec in Heat just yet:

https://review.openstack.org/#/c/97939/

There are a number of other issues noted in that spec which are already
addressed in Heat, but require refactoring in TripleO's templates and
tools, and that work continues.

The point remains: we need something that works now, and doing an
alternate implementation for updates is actually faster than addressing
all of these issues.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [operstack-dev][Congress] congress-server fails

[Rajdeep Dua]

Following command gives an error

./*bin*/*congress*-*server* --config-file etc/congress.conf.sample

Error i am getting is ImportError: No module named
keystonemiddleware.auth_token

Tried installing keystonemiddleware manually, got the following error

$ sudo pip install keystonemiddleware
Downloading/unpacking keystonemiddleware
  Running setup.py egg_info for package keystonemiddleware
[pbr] Reusing existing SOURCES.txt
Requirement already satisfied (use --upgrade to upgrade): Babel>=1.3 in
/usr/local/lib/python2.7/dist-packages (from keystonemiddleware)
Requirement already satisfied (use --upgrade to upgrade): iso8601>=0.1.9 in
/usr/local/lib/python2.7/dist-packages (from keystonemiddleware)
Requirement already satisfied (use --upgrade to upgrade): netaddr>=0.7.6 in
/usr/local/lib/python2.7/dist-packages (from keystonemiddleware)
Downloading/unpacking oslo.config>=1.4.0.0a3 (from keystonemiddleware)
  Could not find a version that satisfies the requirement
oslo.config>=1.4.0.0a3 (from keystonemiddleware) (from versions: )
No distributions matching the version for oslo.config>=1.4.0.0a3 (from
keystonemiddleware)
Storing complete log in /home/hadoop/.pip/pip.log

Thanks
Rajdeep
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

[Jay Pipes]

Hi Li, comments inline.

On 08/08/2014 12:03 AM, Li Ma wrote:

Getting a massive amount of information from data storage to be displayed is
where most of the activity happens in OpenStack. The two activities of reading
data and writing (creating, updating and deleting) data are fundamentally
different.

The optimization for these two opposite database activities can be done by
physically separating the databases that service these two different
activities. All the writes go to database servers, which then replicates the
written data to the database server(s) dedicated to servicing the reads.

Currently, AFAIK, many OpenStack deployment in production try to take
advantage of MySQL (includes Percona or MariaDB) multi-master Galera cluster.
It is possible to design and implement a read/write separation schema
for such a DB cluster.


The above does not really make sense for MySQL Galera/PXC clusters *if 
only Galera nodes are used in the cluster*. Since Galera is 
synchronously replicated, there's no real point in segregating writers 
from readers, IMO. Better to just spread the write AND read load equally 
among all Galera cluster nodes.


However, if you have a Galera cluster that then slaves off to one or 
more standard MySQL slaves, then certainly doing writer/reader 
segregation could be useful, especially for directing readers of 
aggregate or report-type data to the read-only slaves.



Actually, OpenStack has a method for read scalability via defining
master_connection and slave_connection in configuration, but this method
lacks of flexibility due to deciding master or slave in the logical
context(code). It's not transparent for application developer.
As a result, it is not widely used in all the OpenStack projects.

So, I'd like to propose a transparent read/write separation method
for oslo.db that every project may happily takes advantage of it
without any code modification.


I've never seen a writer/reader segregation proxy or middleware piece 
that was properly able to send the "right" reads to the slaves. 
Unfortunately, determining what are the "right" reads to send to the 
slaves is highly application-dependent, since the application knows when 
it can tolerate slave lags.



Moreover, I'd like to put it in the mailing list in advance to
make sure it is acceptable for oslo.db.


I think oslo.db is not the right place for this. I believe the efforts 
that Mike Wilson has been doing in the "slavification" blueprints are 
the more appropriate place to add this slave-aware code.


Best,
-jay


I'd appreciate any comments.

br.
Li Ma


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Congress] congress-server fails to start

[Rajdeep Dua]

Hi All,
command to start the congress-server fails

$ ./bin/congress-server --config-file etc/congress.conf.sample

Error :
ImportError: No module named keystonemiddleware.auth_token

Installing keystonemiddleware manually also fails

$ sudo pip install keystonemiddleware

Could not find a version that satisfies the requirement
oslo.config>=1.4.0.0a3 (from keystonemiddleware) (from versions: )
No distributions matching the version for oslo.config>=1.4.0.0a3 (from
keystonemiddleware)

Thanks
Rajdeep
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

[Steven Hardy]

On Mon, Aug 11, 2014 at 11:20:50AM -0700, Clint Byrum wrote:
> Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:
> > On 11/08/14 10:46, Clint Byrum wrote:
> > > Right now we're stuck with an update that just doesn't work. It isn't
> > > just about update-failure-recovery, which is coming along nicely, but
> > > it is also about the lack of signals to control rebuild, poor support
> > > for addressing machines as groups, and unacceptable performance in
> > > large stacks.
> > 
> > Are there blueprints/bugs filed for all of these issues?
> > 
> 
> Convergnce addresses the poor performance for large stacks in general.
> We also have this:
> 
> https://bugs.launchpad.net/heat/+bug/1306743
> 
> Which shows how slow metadata access can get. I have worked on patches
> but haven't been able to complete them. We made big strides but we are
> at a point where 40 nodes polling Heat every 30s is too much for one CPU
> to handle. When we scaled Heat out onto more CPUs on one box by forking
> we ran into eventlet issues. We also ran into issues because even with
> many processes we can only use one to resolve templates for a single
> stack during update, which was also excessively slow.

Related to this, and a discussion we had recently at the TripleO meetup is
this spec I raised today:

https://review.openstack.org/#/c/113296/

It's following up on the idea that we could potentially address (or at
least mitigate, pending the fully convergence-ified heat) some of these
scalability concerns, if TripleO moves from the one-giant-template model
to a more modular nested-stack/provider model (e.g what Tomas has been
working on)

I've not got into enough detail on that yet to be sure if it's acheivable
for Juno, but it seems initially to be complex-but-doable.

I'd welcome feedback on that idea and how it may fit in with the more
granular convergence-engine model.

Can you link to the eventlet/forking issues bug please?  I thought since
bug #1321303 was fixed that multiple engines and multiple workers should
work OK, and obviously that being true is a precondition to expending
significant effort on the nested stack decoupling plan above.

Steve

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] oslo.concurrency repo review

[Ben Nemec]

On 08/11/2014 01:02 PM, Yuriy Taraday wrote:
> On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow  wrote:
> 
>> One question from me:
>>
>> Will there be later fixes to remove oslo.config dependency/usage from
>> oslo.concurrency?
>>
>> I still don't understand how oslo.concurrency can be used as a library
>> with the configuration being set in a static manner via oslo.config (let's
>> use the example of `lock_path` @ https://github.com/YorikSar/
>> oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). For
>> example:
>>
>> Library X inside application Z uses lockutils (via the nice
>> oslo.concurrency library) and sets the configuration `lock_path` to its
>> desired settings, then library Y (also a user of oslo.concurrency) inside
>> same application Z sets the configuration for `lock_path` to its desired
>> settings. Now both have some unknown set of configuration they have set and
>> when library X (or Y) continues to use lockutils they will be using some
>> mix of configuration (likely some mish mash of settings set by X and Y);
>> perhaps to a `lock_path` that neither actually wants to be able to write
>> to...
>>
>> This doesn't seem like it will end well; and will just cause headaches
>> during debug sessions, testing, integration and more...
>>
>> The same question can be asked about the `set_defaults()` function, how is
>> library Y or X expected to use this (are they?)??
>>
>> I hope one of the later changes is to remove/fix this??
>>
>> Thoughts?
>>
>> -Josh
> 
> 
> I'd be happy to remove lock_path config variable altogether. It's basically
> never used. There are two basic branches in code wrt lock_path:
> - when you provide lock_path argument to lock (and derivative functions),
> file-based lock is used and CONF.lock_path is ignored;
> - when you don't provide lock_path in arguments, semaphore-based lock is
> used and CONF.lock_path is just a prefix for its name (before hashing).
> 
> I wonder if users even set lock_path in their configs as it has almost no
> effect. So I'm all for removing it, but...
> From what I understand, every major change in lockutils drags along a lot
> of headache for everybody (and risk of bugs that would be discovered very
> late). So is such change really worth it? And if so, it will require very
> thorough research of lockutils usage patterns.

Two things lock_path has to stay for: Windows and consumers who require
file-based locking semantics.  Neither of those use cases are trivial to
remove, so IMHO it would not be appropriate to do it as part of the
graduation.  If we were going to alter the API that much it needed to
happen in incubator.

As far as lock_path mismatches, that shouldn't be a problem unless a
consumer is doing something very unwise.  Oslo libs get their
configuration from the application using them, so unless the application
passes two separate conf objects to library X and Y they're both going
to get consistent settings.  If someone _is_ doing that, then I think
it's their responsibility to make sure the options in both config files
are compatible with each other.

-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Infra] Meeting Tuesday August 12th at 19:00 UTC

[Elizabeth K. Joseph]

Hi everyone,

The OpenStack Infrastructure (Infra) team is hosting our weekly
meeting on Tuesday August 12th, at 19:00 UTC in #openstack-meeting

Meeting agenda available here:
https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting (anyone is
welcome to to add agenda items)

Everyone interested in infrastructure and process surrounding
automated testing and deployment is encouraged to attend.

-- 
Elizabeth Krumbach Joseph || Lyz || pleia2
http://www.princessleia.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

[Clint Byrum]

Excerpts from Steven Hardy's message of 2014-08-11 11:40:07 -0700:
> On Mon, Aug 11, 2014 at 11:20:50AM -0700, Clint Byrum wrote:
> > Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:
> > > On 11/08/14 10:46, Clint Byrum wrote:
> > > > Right now we're stuck with an update that just doesn't work. It isn't
> > > > just about update-failure-recovery, which is coming along nicely, but
> > > > it is also about the lack of signals to control rebuild, poor support
> > > > for addressing machines as groups, and unacceptable performance in
> > > > large stacks.
> > > 
> > > Are there blueprints/bugs filed for all of these issues?
> > > 
> > 
> > Convergnce addresses the poor performance for large stacks in general.
> > We also have this:
> > 
> > https://bugs.launchpad.net/heat/+bug/1306743
> > 
> > Which shows how slow metadata access can get. I have worked on patches
> > but haven't been able to complete them. We made big strides but we are
> > at a point where 40 nodes polling Heat every 30s is too much for one CPU
> > to handle. When we scaled Heat out onto more CPUs on one box by forking
> > we ran into eventlet issues. We also ran into issues because even with
> > many processes we can only use one to resolve templates for a single
> > stack during update, which was also excessively slow.
> 
> Related to this, and a discussion we had recently at the TripleO meetup is
> this spec I raised today:
> 
> https://review.openstack.org/#/c/113296/
> 
> It's following up on the idea that we could potentially address (or at
> least mitigate, pending the fully convergence-ified heat) some of these
> scalability concerns, if TripleO moves from the one-giant-template model
> to a more modular nested-stack/provider model (e.g what Tomas has been
> working on)
> 
> I've not got into enough detail on that yet to be sure if it's acheivable
> for Juno, but it seems initially to be complex-but-doable.
> 
> I'd welcome feedback on that idea and how it may fit in with the more
> granular convergence-engine model.
> 
> Can you link to the eventlet/forking issues bug please?  I thought since
> bug #1321303 was fixed that multiple engines and multiple workers should
> work OK, and obviously that being true is a precondition to expending
> significant effort on the nested stack decoupling plan above.
> 

That was the issue. So we fixed that bug, but we never un-reverted
the patch that forks enough engines to use up all the CPU's on a box
by default. That would likely help a lot with metadata access speed
(we could manually do it in TripleO but we tend to push defaults. :)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [TripleO] reverting the HOT migration? // dealing with lockstep changes

[Robert Collins]

Hi, so shortly after the HOT migration landed, we hit
https://bugs.launchpad.net/tripleo/+bug/1354305 which is that on even
quite recently deployed clouds, the migrated templates were just too
new. A partial revert (of just the list_join bit) fixes that, but a
deeper problem emerged which is that stack-update to get from a
non-HOT to HOT template appears broken
(https://bugs.launchpad.net/heat/+bug/1354962).

I think we need to revert the HOT migration today, as forcing a
scorched earth recreation of a cloud is not a great answer for folk
that have deployed versions - its a backwards compat issue.

Its true that our release as of icehouse isn't  really useable, so we
could try to wiggle our way past this one, but I think as the first
real test of our new backwards compat policy, that that would be a
mistake.

What we need to be able to land it again, is some way whereby an
existing cloud can upgrade their undercloud (via stack-update against
the old heat deployed in the seed [today, old heat in the undercloud
itself in future]) and then once that is deployed subsequent templates
can use the new features. We're likely to run into such lockstep
changes in future, so we also need to be able to recognise them in
review / design, and call them out so we can fix them early rather
than deep down the pike.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] usage patterns for oslo.config

[Doug Hellmann]

On Aug 8, 2014, at 7:22 PM, Devananda van der Veen  
wrote:

> On Fri, Aug 8, 2014 at 12:41 PM, Doug Hellmann  wrote:
>> 
>> That’s right. The preferred approach is to put the register_opt() in
>> *runtime* code somewhere before the option will be used. That might be in
>> the constructor for a class that uses an option, for example, as described
>> in
>> http://docs.openstack.org/developer/oslo.config/cfg.html#registering-options
>> 
>> Doug
> 
> Interesting.
> 
> I've been following the prevailing example in Nova, which is to
> register opts at the top of a module, immediately after defining them.
> Is there a situation in which one approach is better than the other?

The approach used in Nova is the “old” way of doing it. It works, but assumes 
that all of the application code is modifying a global configuration object. 
The runtime approach allows you to pass a configuration object to a library, 
which makes it easier to mock the configuration for testing and avoids having 
the configuration options bleed into the public API of the library. We’ve 
started using the runtime approach in new Oslo libraries that have 
configuration options, but changing the implementation in existing application 
code isn’t strictly necessary.

Doug

> 
> Thanks,
> Devananda
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

[Eoghan Glynn]

> Hi Eoghan,
> 
> Thanks for the note below. However, one thing the overview below does not
> cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged. Many
> folks feel that this technology is a viable solution for the problem space
> discussed below.

Great question Brad!

As it happens we've been working closely with Paul Dix (lead
developer of InfluxDB) to ensure that this metrics store would be
usable as a backend driver. That conversation actually kicked off
at the Juno summit in Atlanta, but it really got off the ground
at our mid-cycle meet-up in Paris on in early July.

I wrote a rough strawman version of an InfluxDB driver in advance
of the mid-cycle to frame the discussion, and Paul Dix traveled
to the meet-up so we could have the discussion face-to-face. The
conclusion was that InfluxDB would indeed potentially be a great
fit, modulo some requirements that we identified during the detailed
discussions:

 * shard-space-based retention & backgrounded deletion
 * capability to merge individual timeseries for cross-aggregation
 * backfill-aware downsampling

The InfluxDB folks have committed to implementing those features in
over July and August, and have made concrete progress on that score.

I hope that provides enough detail to answer to your question?

Cheers,
Eoghan

> Thanks,
> 
> Brad
> 
> 
> Brad Topol, Ph.D.
> IBM Distinguished Engineer
> OpenStack
> (919) 543-0646
> Internet: bto...@us.ibm.com
> Assistant: Kendra Witherspoon (919) 254-0680
> 
> 
> 
> From: Eoghan Glynn 
> To: "OpenStack Development Mailing List (not for usage questions)"
> ,
> Date: 08/06/2014 11:17 AM
> Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
> project
> 
> 
> 
> 
> 
> Folks,
> 
> It's come to our attention that some key individuals are not
> fully up-to-date on gnocchi activities, so it being a good and
> healthy thing to ensure we're as communicative as possible about
> our roadmap, I've provided a high-level overview here of our
> thinking. This is intended as a precursor to further discussion
> with the TC.
> 
> Cheers,
> Eoghan
> 
> 
> What gnocchi is:
> ===
> 
> Gnocchi is a separate, but related, project spun up on stackforge
> by Julien Danjou, with the objective of providing efficient
> storage and retrieval of timeseries-oriented data and resource
> representations.
> 
> The goal is to experiment with a potential approach to addressing
> an architectural misstep made in the very earliest days of
> ceilometer, specifically the decision to store snapshots of some
> resource metadata alongside each metric datapoint. The core idea
> is to move to storing datapoints shorn of metadata, and instead
> allow the resource-state timeline to be reconstructed more cheaply
> from much less frequently occurring events (e.g. instance resizes
> or migrations).
> 
> 
> What gnocchi isn't:
> ==
> 
> Gnocchi is not a large-scale under-the-radar rewrite of a core
> OpenStack component along the lines of keystone-lite.
> 
> The change is concentrated on the final data-storage phase of
> the ceilometer pipeline, so will have little initial impact on the
> data-acquiring agents, or on transformation phase.
> 
> We've been totally open at the Atlanta summit and other forums
> about this approach being a multi-cycle effort.
> 
> 
> Why we decided to do it this way:
> 
> 
> The intent behind spinning up a separate project on stackforge
> was to allow the work progress at arms-length from ceilometer,
> allowing normalcy to be maintained on the core project and a
> rapid rate of innovation on gnocchi.
> 
> Note that that the developers primarily contributing to gnocchi
> represent a cross-section of the core team, and there's a regular
> feedback loop in the form of a recurring agenda item at the
> weekly team meeting to avoid the effort becoming silo'd.
> 
> 
> But isn't re-architecting frowned upon?
> ==
> 
> Well, the architecture of other OpenStack projects have also
> under-gone change as the community understanding of the
> implications of prior design decisions has evolved.
> 
> Take for example the move towards nova no-db-compute & the
> unified-object-model in order to address issues in the nova
> architecture that made progress towards rolling upgrades
> unneccessarily difficult.
> 
> The point, in my understanding, is not to avoid doing the
> course-correction where it's deemed necessary. Rather, the
> principle is more that these corrections happen in an open
> and planned way.
> 
> 
> The path forward:
> 
> 
> A subset of the ceilometer community will continue to work on
> gnocchi in parallel with the ceilometer core over the remainder
> of the Juno cycle and into the Kilo timeframe. The goal is to
> have an initial implementation of gnocchi ready for tech preview
> by the end of Juno, and to have the integration/migration/
> co-existence questions addressed in Kilo.
> 

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Vijay Venkatachalam]

Yes, the point was to say "the plugin need not restrict and let driver decide 
what to do with the API".

Even if the call was made to driver instantaneously, I understand, the driver 
might decide to ignore 
first and schedule later. But, if the call is present, there is scope for 
validation. 
Also, the driver might be scheduling an async-api to backend, in which case  
deployment error 
cannot be shown to the user instantaneously.

W.r.t. identifying a provider/driver, how would it be to make tenant the 
default "root" object? 
"tenantid" is already associated with each of these entities, so no additional 
pain. 
For the tenant who wants to override let him specify provider in each of the 
entities.
If you think of this in terms of the UI, let's say if the loadbalancer 
configuration is exposed 
as a single wizard (which has loadbalancer, listener, pool, monitor properties) 
then provider
 is chosen only once. 

Curious question, is flavour framework expected to address this problem?

Thanks,
Vijay V.

-Original Message-
From: Doug Wiegley [mailto:do...@a10networks.com] 
Sent: 11 August 2014 22:02
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver 
interface on every API request"

Hi Sam,

Very true.  I think that Vijay’s objection is that we are currently imposing a 
logical structure on the driver, when it should be a driver decision.  
Certainly, it goes both ways.

And I also agree that the mechanism for returning multiple errors, and the 
ability to specify whether those errors are fatal or not, individually, is 
currently weak.

Doug


On 8/11/14, 10:21 AM, "Samuel Bercovici"  wrote:

>Hi Doug,
>
>In some implementations Driver !== Device. I think this is also true 
>for HA Proxy.
>This might mean that there is a difference between creating a logical 
>object and when there is enough information to actually schedule/place 
>this into a device.
>The ability to express such errors (detecting an error on a logical 
>object after it was created but when it actually get scheduled) should 
>be discussed and addressed anyway.
>
>-Sam.
>
>
>-Original Message-
>From: Doug Wiegley [mailto:do...@a10networks.com]
>Sent: Monday, August 11, 2014 6:55 PM
>To: OpenStack Development Mailing List (not for usage questions)
>Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling 
>driver interface on every API request"
>
>Hi all,
>
>> Validations such as ³timeout > delay² should be performed on the API 
>>level before it reaches the driver.
>For a configuration tree (lb, listeners, pools, etc.), there should be 
>one provider.
>
>You¹re right, but I think the point of Vijay¹s example was to highlight 
>the combo error problem with populating all of the driver objects at 
>once (in short, the driver interface isn¹t well suited to that model.)  
>That his one example can be covered by API validators is irrelevant.  
>Consider a backend that does not support APP_COOKIE¹s, or 
>HTTPS_TERMINATED (but has multiple listeners) instead.  Should the 
>entire load balancer create fail, or should it offer degraded service?  
>Do all drivers have to implement a transaction rollback; wait, the 
>interface makes that very hard.  That¹s his point.  The driver is no 
>longer just glue code between interfaces; it¹s now a mini-object error handler.
>
>
>> Having provider defined in multiple places does not make sense.
>
>Channeling Brandon, who can yell if I get this wrong, the point is not 
>to have a potentially different provider on each object.  It¹s to allow 
>a provider to be assigned when the first object in the tree is created, 
>so that future related objects will always get routed to the same provider.
>Not knowing which provider should get all the objects is why we have to 
>wait until we see a LoadBalancer object.
>
>
>All of this sort of edge case nonsense is because we (the royal we, the 
>community), wanted all load balancer objects to be ³root² objects, even 
>though only one of them is an actual root today, to support 
>many-to-many relationships among all of them, at some future date, 
>without an interface change.  If my bias is showing that I¹m not a fan 
>of adding this complexity for that, I¹m not surprised.
>
>Thanks,
>doug
>
>
>On 8/11/14, 7:57 AM, "Samuel Bercovici"  wrote:
>
>>Hi,
>> 
>>Validations such as ³timeout > delay² should be performed on the API 
>>level before it reaches the driver.
>> 
>>For a configuration tree (lb, listeners, pools, etc.), there should be 
>>one provider.
>>
>>Having provider defined in multiple places does not make sense.
>> 
>> 
>>-San.
>> 
>> 
>>From: Vijay Venkatachalam [mailto:vijay.venkatacha...@citrix.com]
>>
>>Sent: Monday, August 11, 2014 2:43 PM
>>To: OpenStack Development Mailing List
>>(openstack-dev@lists.openstack.org)
>>Subject: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling 
>>driver interface on every API request"
>>
>>
>> 
>>Hi:
>> 
>>

Re: [openstack-dev] [oslo] oslo.concurrency repo review

[Joshua Harlow]

On Mon, Aug 11, 2014 at 11:39 AM, Ben Nemec  
wrote:

On 08/11/2014 01:02 PM, Yuriy Taraday wrote:
 On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow 
 wrote:
 

 One question from me:

 Will there be later fixes to remove oslo.config dependency/usage 
from

 oslo.concurrency?

 I still don't understand how oslo.concurrency can be used as a 
library
 with the configuration being set in a static manner via 
oslo.config (let's

 use the example of `lock_path` @ https://github.com/YorikSar/
 oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
For

 example:

 Library X inside application Z uses lockutils (via the nice
 oslo.concurrency library) and sets the configuration `lock_path` 
to its
 desired settings, then library Y (also a user of oslo.concurrency) 
inside
 same application Z sets the configuration for `lock_path` to its 
desired
 settings. Now both have some unknown set of configuration they 
have set and
 when library X (or Y) continues to use lockutils they will be 
using some
 mix of configuration (likely some mish mash of settings set by X 
and Y);
 perhaps to a `lock_path` that neither actually wants to be able to 
write

 to...

 This doesn't seem like it will end well; and will just cause 
headaches

 during debug sessions, testing, integration and more...

 The same question can be asked about the `set_defaults()` 
function, how is

 library Y or X expected to use this (are they?)??

 I hope one of the later changes is to remove/fix this??

 Thoughts?

 -Josh
 
 
 I'd be happy to remove lock_path config variable altogether. It's 
basically

 never used. There are two basic branches in code wrt lock_path:
 - when you provide lock_path argument to lock (and derivative 
functions),

 file-based lock is used and CONF.lock_path is ignored;
 - when you don't provide lock_path in arguments, semaphore-based 
lock is
 used and CONF.lock_path is just a prefix for its name (before 
hashing).
 
 I wonder if users even set lock_path in their configs as it has 
almost no

 effect. So I'm all for removing it, but...
 From what I understand, every major change in lockutils drags along 
a lot
 of headache for everybody (and risk of bugs that would be 
discovered very
 late). So is such change really worth it? And if so, it will 
require very

 thorough research of lockutils usage patterns.


Two things lock_path has to stay for: Windows and consumers who 
require
file-based locking semantics.  Neither of those use cases are trivial 
to

remove, so IMHO it would not be appropriate to do it as part of the
graduation.  If we were going to alter the API that much it needed to
happen in incubator.


As far as lock_path mismatches, that shouldn't be a problem unless a
consumer is doing something very unwise.  Oslo libs get their
configuration from the application using them, so unless the 
application

passes two separate conf objects to library X and Y they're both going
to get consistent settings.  If someone _is_ doing that, then I think
it's their responsibility to make sure the options in both config 
files

are compatible with each other.


Why would it be assumed they would pass the same settings (how is that 
even possible to know ahead of time? especially if library X pulls in a 
new library ZZ that requires a new configuration setting). For example, 
one directory for `lock_path` may be reasonable for tooz and another 
may be reasonable for taskflow (completly depends on there intended 
usage), it would not likely desireable to have them go to the same 
location. Forcing application Z to know the inner workings of library X 
and library Y (or future unknown library ZZ) is just pushing the 
problem onto the library user, which seems inappropriate and breaks the 
whole point of having abstractions & APIs in the first place... This 
IMHO is part of the problem with having statically set *action at a 
distance* type of configuration, the libraries themselves are not in 
control of their own configuration, which breaks abstractions & APIs 
left and right. If some application Y can go under a library and pull 
the rug out from under it, how is that a reasonable thing to expect the 
library to be able to predict & handle?


This kind of requirement has always made me wonder how other libraries 
(like tooz, or taskflow) actually interact with any of the oslo.* 
libraries in any predicatable way (since those library could be 
interacting with oslo.* libraries that have configuration that can be 
switched out from underneath them, making those libraries have *secret* 
APIs that appear and disappear depending on what used oslo.* library 
was newly added as a dependency and what newly added configuration that 
library sucked in/exposed...).


-Josh




-Ben

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing 

Re: [openstack-dev] [TripleO] reverting the HOT migration? // dealing with lockstep changes

[Dan Prince]

On Tue, 2014-08-12 at 06:58 +1200, Robert Collins wrote:
> Hi, so shortly after the HOT migration landed, we hit
> https://bugs.launchpad.net/tripleo/+bug/1354305 which is that on even
> quite recently deployed clouds, the migrated templates were just too
> new. A partial revert (of just the list_join bit) fixes that, but a
> deeper problem emerged which is that stack-update to get from a
> non-HOT to HOT template appears broken
> (https://bugs.launchpad.net/heat/+bug/1354962).
> 
> I think we need to revert the HOT migration today, as forcing a
> scorched earth recreation of a cloud is not a great answer for folk
> that have deployed versions - its a backwards compat issue.
> 
> Its true that our release as of icehouse isn't  really useable, so we
> could try to wiggle our way past this one, but I think as the first
> real test of our new backwards compat policy, that that would be a
> mistake.

Hmmm. We blocked a good bit of changes to get these HOT templates in so
I hate to see us revert them. Also, It isn't clear to me how much work
it would be to fully support the non-HOT to HOT templates upgrade path.
How much work is this? And is that something we really want to spend
time on instead of all the other things?

Why not just create a branch of the old templates that works on the
existing underclouds? Users would then need to use these templates as a
one-time upgrade step to be able to upgrade to a heat HOT capable
undercloud first.

With regards to the seed...

Would a tool that allowed us to migrate the stateful data from an old
seed to a new seed be a better use of time here? A re-seeder might be a
useful tool to future proof upgrade paths that rely on the software
versions in the seed (Heat etc.) that aren't easily up-gradable.
Packages would help here too, but not everyone uses them...

Dan

> 
> What we need to be able to land it again, is some way whereby an
> existing cloud can upgrade their undercloud (via stack-update against
> the old heat deployed in the seed [today, old heat in the undercloud
> itself in future]) and then once that is deployed subsequent templates
> can use the new features. We're likely to run into such lockstep
> changes in future, so we also need to be able to recognise them in
> review / design, and call them out so we can fix them early rather
> than deep down the pike.
> 
> -Rob
> 



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] oslo.concurrency repo review

[Joshua Harlow]

On Mon, Aug 11, 2014 at 11:02 AM, Yuriy Taraday  
wrote:
On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow  
wrote:

One question from me:

Will there be later fixes to remove oslo.config dependency/usage 
from oslo.concurrency?


I still don't understand how oslo.concurrency can be used as a 
library with the configuration being set in a static manner via 
oslo.config (let's use the example of `lock_path` @ 
https://github.com/YorikSar/oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
For example:


Library X inside application Z uses lockutils (via the nice 
oslo.concurrency library) and sets the configuration `lock_path` to 
its desired settings, then library Y (also a user of 
oslo.concurrency) inside same application Z sets the configuration 
for `lock_path` to its desired settings. Now both have some unknown 
set of configuration they have set and when library X (or Y) 
continues to use lockutils they will be using some mix of 
configuration (likely some mish mash of settings set by X and Y); 
perhaps to a `lock_path` that neither actually wants to be able to 
write to...


This doesn't seem like it will end well; and will just cause 
headaches during debug sessions, testing, integration and more...


The same question can be asked about the `set_defaults()` function, 
how is library Y or X expected to use this (are they?)??


I hope one of the later changes is to remove/fix this??

Thoughts?

-Josh


I'd be happy to remove lock_path config variable altogether. It's 
basically never used. There are two basic branches in code wrt 
lock_path:
- when you provide lock_path argument to lock (and derivative 
functions), file-based lock is used and CONF.lock_path is ignored; 
- when you don't provide lock_path in arguments, semaphore-based lock 
is used and CONF.lock_path is just a prefix for its name (before 
hashing).


Agreed, it just seems confusing (and bad) to have parts of the API come 
in from `CONF.lock_path` (or other `CONF.*` options) and other parts of 
the API come in via function parameters. This just makes understanding 
the API and knowing how to interact with it that much harder (after all 
what is the right way of using XYZ feature when it can be changed via a 
out-of-band *hidden* API call via configuration adjustments under the 
covers?)... This makes it really hard to use oslo.concurrency in 
taskflow (and likely other libraries that would like to consume 
oslo.concurrency, seeing that it will be on pypi, I would expect this 
number to grow...) since taskflow would really appreciate and desire to 
have stable APIs that don't change by some configuration that can be 
set by some party via some out-of-band method (for example some other 
library or program calling `set_defaults()`). This kind of way of using 
an API (half of the settings from config, half of the settings from the 
functions API...) may be ok for applications but it's not IMHO ok for 
libraries (or clients) that want to use oslo.concurrency. 

Hopefully it can be fixed some that it works via both ways? Oslo.db I 
believe made this work better by allowing for configuration to come in 
via a configuration object that can be provided by the user of oslo.db, 
this makes the API that oslo.db exposes strongly tied to the attributes 
& documentation of that object. I still don't think thats perfect 
either since its likely that the documentation for what that objects 
attributes should be is not as update to date or easy to change as 
updating function/method documentation...


I wonder if users even set lock_path in their configs as it has 
almost no effect. So I'm all for removing it, but...
From what I understand, every major change in lockutils drags along a 
lot of headache for everybody (and risk of bugs that would be 
discovered very late). So is such change really worth it? And if so, 
it will require very thorough research of lockutils usage patterns.


Sounds like tech debt to me, it always requires work to make something 
better. Are we the type of community that will avoid changing things 
(for the better) because we fear introducing new bugs that may be found 
along the way? I for one hope that we are not that type of community 
(that type of community will die due to its own *fake* fears...).




--

Kind regards, Yuriy.



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo] oslo.concurrency repo review

[Doug Hellmann]

On Aug 11, 2014, at 3:26 PM, Joshua Harlow  wrote:

> 
> 
> On Mon, Aug 11, 2014 at 11:02 AM, Yuriy Taraday  wrote:
>> On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow  wrote:
>>> One question from me:
>>> Will there be later fixes to remove oslo.config dependency/usage from 
>>> oslo.concurrency?
>>> I still don't understand how oslo.concurrency can be used as a library with 
>>> the configuration being set in a static manner via oslo.config (let's use 
>>> the example of `lock_path` @ 
>>> https://github.com/YorikSar/oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41).
>>>  For example:
>>> Library X inside application Z uses lockutils (via the nice 
>>> oslo.concurrency library) and sets the configuration `lock_path` to its 
>>> desired settings, then library Y (also a user of oslo.concurrency) inside 
>>> same application Z sets the configuration for `lock_path` to its desired 
>>> settings. Now both have some unknown set of configuration they have set and 
>>> when library X (or Y) continues to use lockutils they will be using some 
>>> mix of configuration (likely some mish mash of settings set by X and Y); 
>>> perhaps to a `lock_path` that neither actually wants to be able to write 
>>> to...
>>> This doesn't seem like it will end well; and will just cause headaches 
>>> during debug sessions, testing, integration and more...
>>> The same question can be asked about the `set_defaults()` function, how is 
>>> library Y or X expected to use this (are they?)??
>>> I hope one of the later changes is to remove/fix this??
>>> Thoughts?
>>> -Josh
>> I'd be happy to remove lock_path config variable altogether. It's basically 
>> never used. There are two basic branches in code wrt lock_path:
>> - when you provide lock_path argument to lock (and derivative functions), 
>> file-based lock is used and CONF.lock_path is ignored; - when you don't 
>> provide lock_path in arguments, semaphore-based lock is used and 
>> CONF.lock_path is just a prefix for its name (before hashing).
> 
> Agreed, it just seems confusing (and bad) to have parts of the API come in 
> from `CONF.lock_path` (or other `CONF.*` options) and other parts of the API 
> come in via function parameters. This just makes understanding the API and 
> knowing how to interact with it that much harder (after all what is the right 
> way of using XYZ feature when it can be changed via a out-of-band *hidden* 
> API call via configuration adjustments under the covers?)... This makes it 
> really hard to use oslo.concurrency in taskflow (and likely other libraries 
> that would like to consume oslo.concurrency, seeing that it will be on pypi, 
> I would 

The libraries placed in the oslo namespace are very much NOT meant to be used 
by anything other than OpenStack. They are intended to be the glue layer 
between OpenStack and some other implementation libraries.

oslo.concurrency wraps pylockfile and the plan is to move the actual lock code 
into pylockfile without the oslo.config dependency. That will make pylockfile 
reusable by taskflow and tooz, and the locking stuff in oslo.concurrency a 
smaller API with consistent configuration for use by applications.

> expect this number to grow...) since taskflow would really appreciate and 
> desire to have stable APIs that don't change by some configuration that can 
> be set by some party via some out-of-band method (for example some other 
> library or program calling `set_defaults()`). This kind of way of using an 
> API (half of the settings from config, half of the settings from the 
> functions API...) may be ok for applications but it's not IMHO ok for 
> libraries (or clients) that want to use oslo.concurrency. 
> Hopefully it can be fixed some that it works via both ways? Oslo.db I believe 
> made this work better by allowing for configuration to come in via a 
> configuration object that can be provided by the user of oslo.db, this makes 
> the API that oslo.db exposes strongly tied to the attributes & documentation 
> of that object. I still don't think thats perfect either since its likely 
> that the documentation for what that objects attributes should be is not as 
> update to date or easy to change as updating function/method documentation…

That technique of having the configuration object passed to the oslo library 
will be repeated in the other new libraries we are creating if they already 
depend on configuration settings of some sort. The configuration options are 
not part of the public API of the library, so they and their definitions will 
be hidden from the caller, but the library has to be given a configuration 
object in order to load the settings for itself.

> 
>> I wonder if users even set lock_path in their configs as it has almost no 
>> effect. So I'm all for removing it, but...
>> From what I understand, every major change in lockutils drags along a lot of 
>> headache for everybody (and risk of bugs that would be discovered very 
>> late). So is such change re

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

[Sandy Walsh]

On 8/11/2014 4:22 PM, Eoghan Glynn wrote:
>
>> Hi Eoghan,
>>
>> Thanks for the note below. However, one thing the overview below does not
>> cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged. Many
>> folks feel that this technology is a viable solution for the problem space
>> discussed below.
> Great question Brad!
>
> As it happens we've been working closely with Paul Dix (lead
> developer of InfluxDB) to ensure that this metrics store would be
> usable as a backend driver. That conversation actually kicked off
> at the Juno summit in Atlanta, but it really got off the ground
> at our mid-cycle meet-up in Paris on in early July.
...
>
> The InfluxDB folks have committed to implementing those features in
> over July and August, and have made concrete progress on that score.
>
> I hope that provides enough detail to answer to your question?

I guess it begs the question, if influxdb will do what you want and it's
open source (MIT) as well as commercially supported, how does gnocchi
differentiate?

> Cheers,
> Eoghan
>
>> Thanks,
>>
>> Brad
>>
>>
>> Brad Topol, Ph.D.
>> IBM Distinguished Engineer
>> OpenStack
>> (919) 543-0646
>> Internet: bto...@us.ibm.com
>> Assistant: Kendra Witherspoon (919) 254-0680
>>
>>
>>
>> From: Eoghan Glynn 
>> To: "OpenStack Development Mailing List (not for usage questions)"
>> ,
>> Date: 08/06/2014 11:17 AM
>> Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
>> project
>>
>>
>>
>>
>>
>> Folks,
>>
>> It's come to our attention that some key individuals are not
>> fully up-to-date on gnocchi activities, so it being a good and
>> healthy thing to ensure we're as communicative as possible about
>> our roadmap, I've provided a high-level overview here of our
>> thinking. This is intended as a precursor to further discussion
>> with the TC.
>>
>> Cheers,
>> Eoghan
>>
>>
>> What gnocchi is:
>> ===
>>
>> Gnocchi is a separate, but related, project spun up on stackforge
>> by Julien Danjou, with the objective of providing efficient
>> storage and retrieval of timeseries-oriented data and resource
>> representations.
>>
>> The goal is to experiment with a potential approach to addressing
>> an architectural misstep made in the very earliest days of
>> ceilometer, specifically the decision to store snapshots of some
>> resource metadata alongside each metric datapoint. The core idea
>> is to move to storing datapoints shorn of metadata, and instead
>> allow the resource-state timeline to be reconstructed more cheaply
>> from much less frequently occurring events (e.g. instance resizes
>> or migrations).
>>
>>
>> What gnocchi isn't:
>> ==
>>
>> Gnocchi is not a large-scale under-the-radar rewrite of a core
>> OpenStack component along the lines of keystone-lite.
>>
>> The change is concentrated on the final data-storage phase of
>> the ceilometer pipeline, so will have little initial impact on the
>> data-acquiring agents, or on transformation phase.
>>
>> We've been totally open at the Atlanta summit and other forums
>> about this approach being a multi-cycle effort.
>>
>>
>> Why we decided to do it this way:
>> 
>>
>> The intent behind spinning up a separate project on stackforge
>> was to allow the work progress at arms-length from ceilometer,
>> allowing normalcy to be maintained on the core project and a
>> rapid rate of innovation on gnocchi.
>>
>> Note that that the developers primarily contributing to gnocchi
>> represent a cross-section of the core team, and there's a regular
>> feedback loop in the form of a recurring agenda item at the
>> weekly team meeting to avoid the effort becoming silo'd.
>>
>>
>> But isn't re-architecting frowned upon?
>> ==
>>
>> Well, the architecture of other OpenStack projects have also
>> under-gone change as the community understanding of the
>> implications of prior design decisions has evolved.
>>
>> Take for example the move towards nova no-db-compute & the
>> unified-object-model in order to address issues in the nova
>> architecture that made progress towards rolling upgrades
>> unneccessarily difficult.
>>
>> The point, in my understanding, is not to avoid doing the
>> course-correction where it's deemed necessary. Rather, the
>> principle is more that these corrections happen in an open
>> and planned way.
>>
>>
>> The path forward:
>> 
>>
>> A subset of the ceilometer community will continue to work on
>> gnocchi in parallel with the ceilometer core over the remainder
>> of the Juno cycle and into the Kilo timeframe. The goal is to
>> have an initial implementation of gnocchi ready for tech preview
>> by the end of Juno, and to have the integration/migration/
>> co-existence questions addressed in Kilo.
>>
>> Moving the ceilometer core to using gnocchi will be contingent
>> on it demonstrating the required performance characteristics and
>> providing the semantics needed to support a v3 

Re: [openstack-dev] [oslo] oslo.concurrency repo review

[Ben Nemec]

On 08/11/2014 02:20 PM, Joshua Harlow wrote:
> 
> 
> On Mon, Aug 11, 2014 at 11:39 AM, Ben Nemec  
> wrote:
>> On 08/11/2014 01:02 PM, Yuriy Taraday wrote:
>>>  On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow 
>>>  wrote:
>>>  
  One question from me:

  Will there be later fixes to remove oslo.config dependency/usage 
 from
  oslo.concurrency?

  I still don't understand how oslo.concurrency can be used as a 
 library
  with the configuration being set in a static manner via 
 oslo.config (let's
  use the example of `lock_path` @ https://github.com/YorikSar/
  oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
 For
  example:

  Library X inside application Z uses lockutils (via the nice
  oslo.concurrency library) and sets the configuration `lock_path` 
 to its
  desired settings, then library Y (also a user of oslo.concurrency) 
 inside
  same application Z sets the configuration for `lock_path` to its 
 desired
  settings. Now both have some unknown set of configuration they 
 have set and
  when library X (or Y) continues to use lockutils they will be 
 using some
  mix of configuration (likely some mish mash of settings set by X 
 and Y);
  perhaps to a `lock_path` that neither actually wants to be able to 
 write
  to...

  This doesn't seem like it will end well; and will just cause 
 headaches
  during debug sessions, testing, integration and more...

  The same question can be asked about the `set_defaults()` 
 function, how is
  library Y or X expected to use this (are they?)??

  I hope one of the later changes is to remove/fix this??

  Thoughts?

  -Josh
>>>  
>>>  
>>>  I'd be happy to remove lock_path config variable altogether. It's 
>>> basically
>>>  never used. There are two basic branches in code wrt lock_path:
>>>  - when you provide lock_path argument to lock (and derivative 
>>> functions),
>>>  file-based lock is used and CONF.lock_path is ignored;
>>>  - when you don't provide lock_path in arguments, semaphore-based 
>>> lock is
>>>  used and CONF.lock_path is just a prefix for its name (before 
>>> hashing).
>>>  
>>>  I wonder if users even set lock_path in their configs as it has 
>>> almost no
>>>  effect. So I'm all for removing it, but...
>>>  From what I understand, every major change in lockutils drags along 
>>> a lot
>>>  of headache for everybody (and risk of bugs that would be 
>>> discovered very
>>>  late). So is such change really worth it? And if so, it will 
>>> require very
>>>  thorough research of lockutils usage patterns.
>>
>> Two things lock_path has to stay for: Windows and consumers who 
>> require
>> file-based locking semantics.  Neither of those use cases are trivial 
>> to
>> remove, so IMHO it would not be appropriate to do it as part of the
>> graduation.  If we were going to alter the API that much it needed to
>> happen in incubator.
>>
>>
>> As far as lock_path mismatches, that shouldn't be a problem unless a
>> consumer is doing something very unwise.  Oslo libs get their
>> configuration from the application using them, so unless the 
>> application
>> passes two separate conf objects to library X and Y they're both going
>> to get consistent settings.  If someone _is_ doing that, then I think
>> it's their responsibility to make sure the options in both config 
>> files
>> are compatible with each other.
> 
> Why would it be assumed they would pass the same settings (how is that 
> even possible to know ahead of time? especially if library X pulls in a 
> new library ZZ that requires a new configuration setting). For example, 
> one directory for `lock_path` may be reasonable for tooz and another 
> may be reasonable for taskflow (completly depends on there intended 
> usage), it would not likely desireable to have them go to the same 
> location. 

The only reason I can see that you would want to do this is to avoid
lock name collisions, but I think I'd rather namespace lib locks than
require users to manage multiple lock paths.  Even the one lock path has
been a hassle.

> Forcing application Z to know the inner workings of library X 
> and library Y (or future unknown library ZZ) is just pushing the 
> problem onto the library user, which seems inappropriate and breaks the 
> whole point of having abstractions & APIs in the first place... This 
> IMHO is part of the problem with having statically set *action at a 
> distance* type of configuration, the libraries themselves are not in 
> control of their own configuration, which breaks abstractions & APIs 
> left and right. If some application Y can go under a library and pull 
> the rug out from under it, how is that a reasonable thing to expect the 
> library to be able to predict & handle?

The application doesn't have to, and shouldn't, know about lib options.
 The libraries have a list_opts method

Re: [openstack-dev] [oslo] oslo.concurrency repo review

[Joshua Harlow]

On Mon, Aug 11, 2014 at 12:47 PM, Doug Hellmann  
wrote:


On Aug 11, 2014, at 3:26 PM, Joshua Harlow  
wrote:


 
 
 On Mon, Aug 11, 2014 at 11:02 AM, Yuriy Taraday 
 wrote:
 On Mon, Aug 11, 2014 at 5:44 AM, Joshua Harlow 
 wrote:

 One question from me:
 Will there be later fixes to remove oslo.config dependency/usage 
from oslo.concurrency?
 I still don't understand how oslo.concurrency can be used as a 
library with the configuration being set in a static manner via 
oslo.config (let's use the example of `lock_path` @ 
https://github.com/YorikSar/oslo.concurrency/blob/master/oslo/concurrency/lockutils.py#L41). 
For example:
 Library X inside application Z uses lockutils (via the nice 
oslo.concurrency library) and sets the configuration `lock_path` 
to its desired settings, then library Y (also a user of 
oslo.concurrency) inside same application Z sets the configuration 
for `lock_path` to its desired settings. Now both have some 
unknown set of configuration they have set and when library X (or 
Y) continues to use lockutils they will be using some mix of 
configuration (likely some mish mash of settings set by X and Y); 
perhaps to a `lock_path` that neither actually wants to be able to 
write to...
 This doesn't seem like it will end well; and will just cause 
headaches during debug sessions, testing, integration and more...
 The same question can be asked about the `set_defaults()` 
function, how is library Y or X expected to use this (are they?)??

 I hope one of the later changes is to remove/fix this??
 Thoughts?
 -Josh
 I'd be happy to remove lock_path config variable altogether. It's 
basically never used. There are two basic branches in code wrt 
lock_path:
 - when you provide lock_path argument to lock (and derivative 
functions), file-based lock is used and CONF.lock_path is ignored; 
- when you don't provide lock_path in arguments, semaphore-based 
lock is used and CONF.lock_path is just a prefix for its name 
(before hashing).
 
 Agreed, it just seems confusing (and bad) to have parts of the API 
come in from `CONF.lock_path` (or other `CONF.*` options) and other 
parts of the API come in via function parameters. This just makes 
understanding the API and knowing how to interact with it that much 
harder (after all what is the right way of using XYZ feature when it 
can be changed via a out-of-band *hidden* API call via configuration 
adjustments under the covers?)... This makes it really hard to use 
oslo.concurrency in taskflow (and likely other libraries that would 
like to consume oslo.concurrency, seeing that it will be on pypi, I 
would 


The libraries placed in the oslo namespace are very much NOT meant to 
be used by anything other than OpenStack. They are intended to be the 
glue layer between OpenStack and some other implementation libraries.


oslo.concurrency wraps pylockfile and the plan is to move the actual 
lock code into pylockfile without the oslo.config dependency. That 
will make pylockfile reusable by taskflow and tooz, and the locking 
stuff in oslo.concurrency a smaller API with consistent configuration 
for use by applications.


Sounds great, I've been wondering why 
https://github.com/stackforge/tooz/commit/f3e11e40f9871f8328 
happened/merged (maybe it should be changed?). I see that 
https://review.openstack.org/#/c/102202/ merged so that's good news and 
hopefully makes the underlying lockutils functionality more useful to 
outside of openstack users in the near-term future (which includes 
taskflow, being that it is being used in & outside openstack by various 
entities).





 expect this number to grow...) since taskflow would really 
appreciate and desire to have stable APIs that don't change by some 
configuration that can be set by some party via some out-of-band 
method (for example some other library or program calling 
`set_defaults()`). This kind of way of using an API (half of the 
settings from config, half of the settings from the functions 
API...) may be ok for applications but it's not IMHO ok for 
libraries (or clients) that want to use oslo.concurrency. 
 Hopefully it can be fixed some that it works via both ways? Oslo.db 
I believe made this work better by allowing for configuration to 
come in via a configuration object that can be provided by the user 
of oslo.db, this makes the API that oslo.db exposes strongly tied to 
the attributes & documentation of that object. I still don't think 
thats perfect either since its likely that the documentation for 
what that objects attributes should be is not as update to date or 
easy to change as updating function/method documentation…


That technique of having the configuration object passed to the oslo 
library will be repeated in the other new libraries we are creating 
if they already depend on configuration settings of some sort. The 
configuration options are not part of the public API of the library, 
so they and their definitions will be hidden from the caller, but the 
library has to b

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Eugene Nikanorov]

Hi folks,

That actually going in opposite direction to what flavor framework is
trying to do (and for dispatching it's doing the same as providers). REST
call dispatching should really go via the root object.

I don't quite get the issue with health monitors. If HM is incorrectly
configured prior to association with a pool - API layer should handle that.
I don't think driver implementations should be different at constraints to
HM parameters.

So I'm -1 on adding provider (or flavor) to each entity. After all, it
looks just like data denormalization which actually will affect lots of API
aspects in negative way.

Thanks,
Eugene.



On Mon, Aug 11, 2014 at 11:20 PM, Vijay Venkatachalam <
vijay.venkatacha...@citrix.com> wrote:

>
> Yes, the point was to say "the plugin need not restrict and let driver
> decide what to do with the API".
>
> Even if the call was made to driver instantaneously, I understand, the
> driver might decide to ignore
> first and schedule later. But, if the call is present, there is scope for
> validation.
> Also, the driver might be scheduling an async-api to backend, in which
> case  deployment error
> cannot be shown to the user instantaneously.
>
> W.r.t. identifying a provider/driver, how would it be to make tenant the
> default "root" object?
> "tenantid" is already associated with each of these entities, so no
> additional pain.
> For the tenant who wants to override let him specify provider in each of
> the entities.
> If you think of this in terms of the UI, let's say if the loadbalancer
> configuration is exposed
> as a single wizard (which has loadbalancer, listener, pool, monitor
> properties) then provider
>  is chosen only once.
>
> Curious question, is flavour framework expected to address this problem?
>
> Thanks,
> Vijay V.
>
> -Original Message-
> From: Doug Wiegley [mailto:do...@a10networks.com]
> Sent: 11 August 2014 22:02
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling
> driver interface on every API request"
>
> Hi Sam,
>
> Very true.  I think that Vijay’s objection is that we are currently
> imposing a logical structure on the driver, when it should be a driver
> decision.  Certainly, it goes both ways.
>
> And I also agree that the mechanism for returning multiple errors, and the
> ability to specify whether those errors are fatal or not, individually, is
> currently weak.
>
> Doug
>
>
> On 8/11/14, 10:21 AM, "Samuel Bercovici"  wrote:
>
> >Hi Doug,
> >
> >In some implementations Driver !== Device. I think this is also true
> >for HA Proxy.
> >This might mean that there is a difference between creating a logical
> >object and when there is enough information to actually schedule/place
> >this into a device.
> >The ability to express such errors (detecting an error on a logical
> >object after it was created but when it actually get scheduled) should
> >be discussed and addressed anyway.
> >
> >-Sam.
> >
> >
> >-Original Message-
> >From: Doug Wiegley [mailto:do...@a10networks.com]
> >Sent: Monday, August 11, 2014 6:55 PM
> >To: OpenStack Development Mailing List (not for usage questions)
> >Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling
> >driver interface on every API request"
> >
> >Hi all,
> >
> >> Validations such as ³timeout > delay² should be performed on the API
> >>level before it reaches the driver.
> >For a configuration tree (lb, listeners, pools, etc.), there should be
> >one provider.
> >
> >You¹re right, but I think the point of Vijay¹s example was to highlight
> >the combo error problem with populating all of the driver objects at
> >once (in short, the driver interface isn¹t well suited to that model.)
> >That his one example can be covered by API validators is irrelevant.
> >Consider a backend that does not support APP_COOKIE¹s, or
> >HTTPS_TERMINATED (but has multiple listeners) instead.  Should the
> >entire load balancer create fail, or should it offer degraded service?
> >Do all drivers have to implement a transaction rollback; wait, the
> >interface makes that very hard.  That¹s his point.  The driver is no
> >longer just glue code between interfaces; it¹s now a mini-object error
> handler.
> >
> >
> >> Having provider defined in multiple places does not make sense.
> >
> >Channeling Brandon, who can yell if I get this wrong, the point is not
> >to have a potentially different provider on each object.  It¹s to allow
> >a provider to be assigned when the first object in the tree is created,
> >so that future related objects will always get routed to the same
> provider.
> >Not knowing which provider should get all the objects is why we have to
> >wait until we see a LoadBalancer object.
> >
> >
> >All of this sort of edge case nonsense is because we (the royal we, the
> >community), wanted all load balancer objects to be ³root² objects, even
> >though only one of them is an actual root today, to support
> >many-to-

Re: [openstack-dev] Which program for Rally

[Matthew Treinish]

I apologize for the delay in my response to this thread, between travelling
and having a stuck 'a' key on my laptop this is the earliest I could respond.
I opted for a separate branch on this thread to summarize my views and I'll
respond inline later on some of the previous discussion.

On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
> Hi everyone,
> 
> At the TC meeting yesterday we discussed Rally program request and
> incubation request. We quickly dismissed the incubation request, as
> Rally appears to be able to live happily on top of OpenStack and would
> benefit from having a release cycle decoupled from the OpenStack
> "integrated release".
> 
> That leaves the question of the program. OpenStack programs are created
> by the Technical Committee, to bless existing efforts and teams that are
> considered *essential* to the production of the "OpenStack" integrated
> release and the completion of the OpenStack project mission. There are 3
> ways to look at Rally and official programs at this point:
> 
> 1. Rally as an essential QA tool
> Performance testing (and especially performance regression testing) is
> an essential QA function, and a feature that Rally provides. If the QA
> team is happy to use Rally to fill that function, then Rally can
> obviously be adopted by the (already-existing) QA program. That said,
> that would put Rally under the authority of the QA PTL, and that raises
> a few questions due to the current architecture of Rally, which is more
> product-oriented. There needs to be further discussion between the QA
> core team and the Rally team to see how that could work and if that
> option would be acceptable for both sides.

So ideally this is where Rally would belong, the scope of what Rally is
attempting to do is definitely inside the scope of the QA program. I don't see
any reason why that isn't the case. The issue is with what Rally is in it's
current form. It's scope is too large and monolithic, and it duplicates much of
the functionality we either already have or need in current QA or Infra
projects. But, nothing in Rally is designed to be used outside of it. I actually
feel pretty strongly that in it's current form Rally should *not* be a part of
any OpenStack program.

All of the points Sean was making in the other branch on this thread (which I'll
probably respond to later) are a huge concerns I share with Rally. He basically
summarized most of my views on the topic, so I'll try not to rewrite everything.
But, the fact that all of this duplicate functionality was implemented in a
completely separate manner which is Rally specific and can't really be used
unless all of Rally is used is of a large concern. What I think the path
forward here is to have both QA and Rally work together on getting common
functionality that is re-usable and shareable. Additionally, I have some
concerns over the methodology that Rally uses for it's performance measurement.
But, I'll table that discussion because I think it would partially derail this
discussion.

So one open question is long-term where would this leave Rally if we want to
bring it in under the QA program. (after splitting up the functionality to more
conducive with all our existing tools and projects) The one thing Rally does
here which we don't have an analogous solution for is, for lack of better term,
the post processing layer. The part that generates the performs the analysis on
the collected data and generates the graphs. That is something that we'll have
an eventually need for and that is something that we can work on turning rally
into as we migrate everything to actually work together.

There are probably also other parts of Rally which don't fit into an existing
QA program project, (or the QA program in general) and in those cases we
probably should split them off as smaller projects to implement that bit. For
example, the SLA stuff Rally has that probably should be a separate entity as
well, but I'm unsure if that fits under QA program.

My primary concern is the timing for doing all of this work. We're approaching
J-3 and honestly this feels like it would take the better part of an entire
cycle to analyze, plan, and then implement. Starting an analysis of how to do
all of the work at this point I feel would just distract everyone from
completing our dev goals for the cycle. Probably the Rally team, if they want
to move forward here, should start the analysis of this structural split and we
can all pick this up together post-juno.

> 
> 2. Rally as an essential operator tool
> Regular benchmarking of OpenStack deployments is a best practice for
> cloud operators, and a feature that Rally provides. With a bit of a
> stretch, we could consider that benchmarking is essential to the
> completion of the OpenStack project mission. That program could one day
> evolve to include more such "operations best practices" tools. In
> addition to the slight stretch already mentioned, one concern here is
> that we still w

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

[Eoghan Glynn]

> On 8/11/2014 4:22 PM, Eoghan Glynn wrote:
> >
> >> Hi Eoghan,
> >>
> >> Thanks for the note below. However, one thing the overview below does not
> >> cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged.
> >> Many
> >> folks feel that this technology is a viable solution for the problem space
> >> discussed below.
> > Great question Brad!
> >
> > As it happens we've been working closely with Paul Dix (lead
> > developer of InfluxDB) to ensure that this metrics store would be
> > usable as a backend driver. That conversation actually kicked off
> > at the Juno summit in Atlanta, but it really got off the ground
> > at our mid-cycle meet-up in Paris on in early July.
> ...
> >
> > The InfluxDB folks have committed to implementing those features in
> > over July and August, and have made concrete progress on that score.
> >
> > I hope that provides enough detail to answer to your question?
> 
> I guess it begs the question, if influxdb will do what you want and it's
> open source (MIT) as well as commercially supported, how does gnocchi
> differentiate?

Hi Sandy,

One of the ideas behind gnocchi is to combine resource representation
and timeseries-oriented storage of metric data, providing an efficient
and convenient way to query for metric data associated with individual
resources.

Also, having an API layered above the storage driver avoids locking in
directly with a particular metrics-oriented DB, allowing for the
potential to support multiple storage driver options (e.g. to choose
between a canonical implementation based on Swift, an InfluxDB driver,
and an OpenTSDB driver, say).

A less compelling reason would be to provide a well-defined hook point
to innovate with aggregation/analytic logic not supported natively
in the underlying drivers (e.g. period-spanning statistics such as
exponentially-weighted moving average or even Holt-Winters).

Cheers,
Eoghan

 
> > Cheers,
> > Eoghan
> >
> >> Thanks,
> >>
> >> Brad
> >>
> >>
> >> Brad Topol, Ph.D.
> >> IBM Distinguished Engineer
> >> OpenStack
> >> (919) 543-0646
> >> Internet: bto...@us.ibm.com
> >> Assistant: Kendra Witherspoon (919) 254-0680
> >>
> >>
> >>
> >> From: Eoghan Glynn 
> >> To: "OpenStack Development Mailing List (not for usage questions)"
> >> ,
> >> Date: 08/06/2014 11:17 AM
> >> Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
> >> project
> >>
> >>
> >>
> >>
> >>
> >> Folks,
> >>
> >> It's come to our attention that some key individuals are not
> >> fully up-to-date on gnocchi activities, so it being a good and
> >> healthy thing to ensure we're as communicative as possible about
> >> our roadmap, I've provided a high-level overview here of our
> >> thinking. This is intended as a precursor to further discussion
> >> with the TC.
> >>
> >> Cheers,
> >> Eoghan
> >>
> >>
> >> What gnocchi is:
> >> ===
> >>
> >> Gnocchi is a separate, but related, project spun up on stackforge
> >> by Julien Danjou, with the objective of providing efficient
> >> storage and retrieval of timeseries-oriented data and resource
> >> representations.
> >>
> >> The goal is to experiment with a potential approach to addressing
> >> an architectural misstep made in the very earliest days of
> >> ceilometer, specifically the decision to store snapshots of some
> >> resource metadata alongside each metric datapoint. The core idea
> >> is to move to storing datapoints shorn of metadata, and instead
> >> allow the resource-state timeline to be reconstructed more cheaply
> >> from much less frequently occurring events (e.g. instance resizes
> >> or migrations).
> >>
> >>
> >> What gnocchi isn't:
> >> ==
> >>
> >> Gnocchi is not a large-scale under-the-radar rewrite of a core
> >> OpenStack component along the lines of keystone-lite.
> >>
> >> The change is concentrated on the final data-storage phase of
> >> the ceilometer pipeline, so will have little initial impact on the
> >> data-acquiring agents, or on transformation phase.
> >>
> >> We've been totally open at the Atlanta summit and other forums
> >> about this approach being a multi-cycle effort.
> >>
> >>
> >> Why we decided to do it this way:
> >> 
> >>
> >> The intent behind spinning up a separate project on stackforge
> >> was to allow the work progress at arms-length from ceilometer,
> >> allowing normalcy to be maintained on the core project and a
> >> rapid rate of innovation on gnocchi.
> >>
> >> Note that that the developers primarily contributing to gnocchi
> >> represent a cross-section of the core team, and there's a regular
> >> feedback loop in the form of a recurring agenda item at the
> >> weekly team meeting to avoid the effort becoming silo'd.
> >>
> >>
> >> But isn't re-architecting frowned upon?
> >> ==
> >>
> >> Well, the architecture of other OpenStack projects have also
> >> under-gone change as the community understanding of the
> >> implications of 

Re: [openstack-dev] [Congress] congress-server fails to start

[Peter Balland]

Hi Rajdeep,

What version of pip are you running?  Please try installing the latest version 
(https://pip.pypa.io/en/latest/installing.html) and run 'sudo pip install -r 
requirements.txt'.

- Peter

From: Rajdeep Dua mailto:rajdeep@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, August 11, 2014 at 11:27 AM
To: "OpenStack Development Mailing List (not for usage questions)" 
mailto:openstack-dev@lists.openstack.org>>
Subject: [openstack-dev] [Congress] congress-server fails to start

Hi All,
command to start the congress-server fails

$ ./bin/congress-server --config-file etc/congress.conf.sample

Error :
ImportError: No module named keystonemiddleware.auth_token

Installing keystonemiddleware manually also fails

$ sudo pip install keystonemiddleware

Could not find a version that satisfies the requirement oslo.config>=1.4.0.0a3 
(from keystonemiddleware) (from versions: )
No distributions matching the version for oslo.config>=1.4.0.0a3 (from 
keystonemiddleware)

Thanks
Rajdeep
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO][heat] a small experiment with Ansible in TripleO

[Zane Bitter]

On 11/08/14 14:49, Clint Byrum wrote:

Excerpts from Steven Hardy's message of 2014-08-11 11:40:07 -0700:

On Mon, Aug 11, 2014 at 11:20:50AM -0700, Clint Byrum wrote:

Excerpts from Zane Bitter's message of 2014-08-11 08:16:56 -0700:

On 11/08/14 10:46, Clint Byrum wrote:

Right now we're stuck with an update that just doesn't work. It isn't
just about update-failure-recovery, which is coming along nicely, but
it is also about the lack of signals to control rebuild, poor support
for addressing machines as groups, and unacceptable performance in
large stacks.


Are there blueprints/bugs filed for all of these issues?



Convergnce addresses the poor performance for large stacks in general.
We also have this:

https://bugs.launchpad.net/heat/+bug/1306743

Which shows how slow metadata access can get. I have worked on patches
but haven't been able to complete them. We made big strides but we are
at a point where 40 nodes polling Heat every 30s is too much for one CPU


This sounds like the same figure I heard at the design summit; did the 
DB call optimisation work that Steve Baker did immediately after that 
not have any effect?



to handle. When we scaled Heat out onto more CPUs on one box by forking
we ran into eventlet issues. We also ran into issues because even with
many processes we can only use one to resolve templates for a single
stack during update, which was also excessively slow.


Related to this, and a discussion we had recently at the TripleO meetup is
this spec I raised today:

https://review.openstack.org/#/c/113296/

It's following up on the idea that we could potentially address (or at
least mitigate, pending the fully convergence-ified heat) some of these
scalability concerns, if TripleO moves from the one-giant-template model
to a more modular nested-stack/provider model (e.g what Tomas has been
working on)

I've not got into enough detail on that yet to be sure if it's acheivable
for Juno, but it seems initially to be complex-but-doable.

I'd welcome feedback on that idea and how it may fit in with the more
granular convergence-engine model.

Can you link to the eventlet/forking issues bug please?  I thought since
bug #1321303 was fixed that multiple engines and multiple workers should
work OK, and obviously that being true is a precondition to expending
significant effort on the nested stack decoupling plan above.



That was the issue. So we fixed that bug, but we never un-reverted
the patch that forks enough engines to use up all the CPU's on a box
by default. That would likely help a lot with metadata access speed
(we could manually do it in TripleO but we tend to push defaults. :)


Right, and we decided we wouldn't because it's wrong to do that to 
people by default. In some cases the optimal running configuration for 
TripleO will differ from the friendliest out-of-the-box configuration 
for Heat users in general, and in those cases - of which this is one - 
TripleO will need to specify the configuration.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Which program for Rally

[David Kranz]

On 08/11/2014 04:21 PM, Matthew Treinish wrote:


I apologize for the delay in my response to this thread, between 
travelling
and having a stuck 'a' key on my laptop this is the earliest I could 
respond.
I opted for a separate branch on this thread to summarize my views and 
I'll

respond inline later on some of the previous discussion.

On Wed, Aug 06, 2014 at 12:30:35PM +0200, Thierry Carrez wrote:
> Hi everyone,
>
> At the TC meeting yesterday we discussed Rally program request and
> incubation request. We quickly dismissed the incubation request, as
> Rally appears to be able to live happily on top of OpenStack and would
> benefit from having a release cycle decoupled from the OpenStack
> "integrated release".
>
> That leaves the question of the program. OpenStack programs are created
> by the Technical Committee, to bless existing efforts and teams that are
> considered *essential* to the production of the "OpenStack" integrated
> release and the completion of the OpenStack project mission. There are 3
> ways to look at Rally and official programs at this point:
>
> 1. Rally as an essential QA tool
> Performance testing (and especially performance regression testing) is
> an essential QA function, and a feature that Rally provides. If the QA
> team is happy to use Rally to fill that function, then Rally can
> obviously be adopted by the (already-existing) QA program. That said,
> that would put Rally under the authority of the QA PTL, and that raises
> a few questions due to the current architecture of Rally, which is more
> product-oriented. There needs to be further discussion between the QA
> core team and the Rally team to see how that could work and if that
> option would be acceptable for both sides.

So ideally this is where Rally would belong, the scope of what Rally is
attempting to do is definitely inside the scope of the QA program. I 
don't see
any reason why that isn't the case. The issue is with what Rally is in 
it's
current form. It's scope is too large and monolithic, and it 
duplicates much of

the functionality we either already have or need in current QA or Infra
projects. But, nothing in Rally is designed to be used outside of it. 
I actually
feel pretty strongly that in it's current form Rally should *not* be a 
part of

any OpenStack program.

All of the points Sean was making in the other branch on this thread 
(which I'll
probably respond to later) are a huge concerns I share with Rally. He 
basically
summarized most of my views on the topic, so I'll try not to rewrite 
everything.
But, the fact that all of this duplicate functionality was implemented 
in a
completely separate manner which is Rally specific and can't really be 
used

unless all of Rally is used is of a large concern. What I think the path
forward here is to have both QA and Rally work together on getting common
functionality that is re-usable and shareable. Additionally, I have some
concerns over the methodology that Rally uses for it's performance 
measurement.
But, I'll table that discussion because I think it would partially 
derail this

discussion.

So one open question is long-term where would this leave Rally if we 
want to
bring it in under the QA program. (after splitting up the 
functionality to more
conducive with all our existing tools and projects) The one thing 
Rally does
here which we don't have an analogous solution for is, for lack of 
better term,
the post processing layer. The part that generates the performs the 
analysis on
the collected data and generates the graphs. That is something that 
we'll have
an eventually need for and that is something that we can work on 
turning rally

into as we migrate everything to actually work together.

There are probably also other parts of Rally which don't fit into an 
existing

QA program project, (or the QA program in general) and in those cases we
probably should split them off as smaller projects to implement that 
bit. For
example, the SLA stuff Rally has that probably should be a separate 
entity as

well, but I'm unsure if that fits under QA program.

My primary concern is the timing for doing all of this work. We're 
approaching
J-3 and honestly this feels like it would take the better part of an 
entire
cycle to analyze, plan, and then implement. Starting an analysis of 
how to do

all of the work at this point I feel would just distract everyone from
completing our dev goals for the cycle. Probably the Rally team, if 
they want
to move forward here, should start the analysis of this structural 
split and we

can all pick this up together post-juno.

>
> 2. Rally as an essential operator tool
> Regular benchmarking of OpenStack deployments is a best practice for
> cloud operators, and a feature that Rally provides. With a bit of a
> stretch, we could consider that benchmarking is essential to the
> completion of the OpenStack project mission. That program could one day
> evolve to include more such "operations best practices" tools. In
> 

Re: [openstack-dev] [Neutron][LBaaS] Continuing on "Calling driver interface on every API request"

[Brandon Logan]

Hi Eugene,
An example of the HM issue (and really this can happen with any entity)
is if the driver the API sends the configuration to does not actually
support the value of an attribute.

For example: Provider A support PING health monitor type, Provider B
does not.  API allows the PING health monitor type to go through.  Once
a load balancer has been linked with that health monitor and the
LoadBalancer chose to use Provider B, that entire configuration is then
sent to the driver.  The driver errors out not on the LoadBalancer
create, but on the health monitor create.

I think that's the issue.

Thanks,
Brandon

On Tue, 2014-08-12 at 00:17 +0400, Eugene Nikanorov wrote:
> Hi folks,
> 
> 
> That actually going in opposite direction to what flavor framework is
> trying to do (and for dispatching it's doing the same as providers).
> REST call dispatching should really go via the root object.
> 
> 
> I don't quite get the issue with health monitors. If HM is incorrectly
> configured prior to association with a pool - API layer should handle
> that.
> I don't think driver implementations should be different at
> constraints to HM parameters.
> 
> 
> So I'm -1 on adding provider (or flavor) to each entity. After all, it
> looks just like data denormalization which actually will affect lots
> of API aspects in negative way.
> 
> 
> Thanks,
> Eugene.
> 
> 
> 
> 
> On Mon, Aug 11, 2014 at 11:20 PM, Vijay Venkatachalam
>  wrote:
> 
> Yes, the point was to say "the plugin need not restrict and
> let driver decide what to do with the API".
> 
> Even if the call was made to driver instantaneously, I
> understand, the driver might decide to ignore
> first and schedule later. But, if the call is present, there
> is scope for validation.
> Also, the driver might be scheduling an async-api to backend,
> in which case  deployment error
> cannot be shown to the user instantaneously.
> 
> W.r.t. identifying a provider/driver, how would it be to make
> tenant the default "root" object?
> "tenantid" is already associated with each of these entities,
> so no additional pain.
> For the tenant who wants to override let him specify provider
> in each of the entities.
> If you think of this in terms of the UI, let's say if the
> loadbalancer configuration is exposed
> as a single wizard (which has loadbalancer, listener, pool,
> monitor properties) then provider
>  is chosen only once.
> 
> Curious question, is flavour framework expected to address
> this problem?
> 
> Thanks,
> Vijay V.
> 
> -Original Message-
> From: Doug Wiegley [mailto:do...@a10networks.com]
> 
> Sent: 11 August 2014 22:02
> To: OpenStack Development Mailing List (not for usage
> questions)
> Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on
> "Calling driver interface on every API request"
> 
> Hi Sam,
> 
> Very true.  I think that Vijay’s objection is that we are
> currently imposing a logical structure on the driver, when it
> should be a driver decision.  Certainly, it goes both ways.
> 
> And I also agree that the mechanism for returning multiple
> errors, and the ability to specify whether those errors are
> fatal or not, individually, is currently weak.
> 
> Doug
> 
> 
> On 8/11/14, 10:21 AM, "Samuel Bercovici" 
> wrote:
> 
> >Hi Doug,
> >
> >In some implementations Driver !== Device. I think this is
> also true
> >for HA Proxy.
> >This might mean that there is a difference between creating a
> logical
> >object and when there is enough information to actually
> schedule/place
> >this into a device.
> >The ability to express such errors (detecting an error on a
> logical
> >object after it was created but when it actually get
> scheduled) should
> >be discussed and addressed anyway.
> >
> >-Sam.
> >
> >
> >-Original Message-
> >From: Doug Wiegley [mailto:do...@a10networks.com]
> >Sent: Monday, August 11, 2014 6:55 PM
> >To: OpenStack Development Mailing List (not for usage
> questions)
> >Subject: Re: [openstack-dev] [Neutron][LBaaS] Continuing on
> "Calling
> >driver interface on every API request"
> >
> >Hi all,
> >
> >> Validations such as ³timeout > delay² should be performed
> on the API
> >>level before it reaches the driver.
> >For a configuration tree (lb, listeners, pools, etc.), there
> should be
> >one p

Re: [openstack-dev] [TripleO][Nova][Neutron] multiple hypervisors on one compute host - neutron agent and compute hostnames

[Joe Gordon]

On Tue, Aug 5, 2014 at 4:17 PM, Robert Collins 
wrote:

> Hi!
>
> James has run into an issue implementing the multi-hypervisor spec
> (
> http://git.openstack.org/cgit/openstack/tripleo-specs/tree/specs/juno/tripleo-juno-deploy-cloud-hypervisor-type.rst
> )
> which we're hoping to use to reduce infrastructure overheads by
> deploying OpenStack control plane services in VMs, without requiring
> dedicated VM hypervisor machines in the deploy cloud.
>
> The issue we've hit is that the Neutron messages for VIF plugging are
> sent to the Neutron agent with an exactly matching hostname to the
> Nova-compute process. However, we have unique hostnames for the
> nova-compute processes on one machine (one for -kvm, one for -docker,
> one for -ironic etc) for a variety of reasons: so we can see if all
> the processes are up, so that we don't get messages for the wrong
> process from nova-api etc.
>

So you are running multiple nova-computes on a single node? This goes
against the model that nova is operating under. Instead of hacking a
workaround, if we think this is a use case nova/openstack should support,
why not have that discussion before deciding that the best solution is a
shallow patch.



>
> I think a reasonable step might be to allow the agent host option to
> be a list - e.g.
>
>  [DEFAULT]
>  hosts={{nova.compute_hostname}}-libvirt,{{nova.compute_hostname}}-docker
>
> we'd just make it listen to all the nova-compute hostnames we may have
> on the machine.
> That seems like a fairly shallow patch to me: add a new hosts option
> with no default, change the code to listen to N queues when hosts is
> set, and to report state N times as well (for consistency).
> Alternatively, with a DB migration, we could record N hosts against
> one agent status.
>
> Alternatively we could run N ovs-agents on one machine (with a
> separate integration bridge each), but I worry that we'd encounter
> unexpected cross-chatter between them on things like external bridge
> flows.
>
> Thoughts?
>
> For now, we're going to have to run with a limitation of only one
> vif-plugging hypervisor type per machine - we'll make the agent
> hostname match that of the nova compute that needs VIFs plugged ;)
>
> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

[Sandy Walsh]

On 8/11/2014 5:29 PM, Eoghan Glynn wrote:
>
>> On 8/11/2014 4:22 PM, Eoghan Glynn wrote:
 Hi Eoghan,

 Thanks for the note below. However, one thing the overview below does not
 cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged.
 Many
 folks feel that this technology is a viable solution for the problem space
 discussed below.
>>> Great question Brad!
>>>
>>> As it happens we've been working closely with Paul Dix (lead
>>> developer of InfluxDB) to ensure that this metrics store would be
>>> usable as a backend driver. That conversation actually kicked off
>>> at the Juno summit in Atlanta, but it really got off the ground
>>> at our mid-cycle meet-up in Paris on in early July.
>> ...
>>> The InfluxDB folks have committed to implementing those features in
>>> over July and August, and have made concrete progress on that score.
>>>
>>> I hope that provides enough detail to answer to your question?
>> I guess it begs the question, if influxdb will do what you want and it's
>> open source (MIT) as well as commercially supported, how does gnocchi
>> differentiate?
> Hi Sandy,
>
> One of the ideas behind gnocchi is to combine resource representation
> and timeseries-oriented storage of metric data, providing an efficient
> and convenient way to query for metric data associated with individual
> resources.

Doesn't InfluxDB do the same?

>
> Also, having an API layered above the storage driver avoids locking in
> directly with a particular metrics-oriented DB, allowing for the
> potential to support multiple storage driver options (e.g. to choose
> between a canonical implementation based on Swift, an InfluxDB driver,
> and an OpenTSDB driver, say).
Right, I'm not suggesting to remove the storage abstraction layer. I'm
just curious what gnocchi does better/different than InfluxDB?

Or, am I missing the objective here and gnocchi is the abstraction layer
and not an influxdb alternative? If so, my apologies for the confusion.

> A less compelling reason would be to provide a well-defined hook point
> to innovate with aggregation/analytic logic not supported natively
> in the underlying drivers (e.g. period-spanning statistics such as
> exponentially-weighted moving average or even Holt-Winters).
> Cheers,
> Eoghan
>
>  
>>> Cheers,
>>> Eoghan
>>>
 Thanks,

 Brad


 Brad Topol, Ph.D.
 IBM Distinguished Engineer
 OpenStack
 (919) 543-0646
 Internet: bto...@us.ibm.com
 Assistant: Kendra Witherspoon (919) 254-0680



 From: Eoghan Glynn 
 To: "OpenStack Development Mailing List (not for usage questions)"
 ,
 Date: 08/06/2014 11:17 AM
 Subject: [openstack-dev] [tc][ceilometer] Some background on the gnocchi
 project





 Folks,

 It's come to our attention that some key individuals are not
 fully up-to-date on gnocchi activities, so it being a good and
 healthy thing to ensure we're as communicative as possible about
 our roadmap, I've provided a high-level overview here of our
 thinking. This is intended as a precursor to further discussion
 with the TC.

 Cheers,
 Eoghan


 What gnocchi is:
 ===

 Gnocchi is a separate, but related, project spun up on stackforge
 by Julien Danjou, with the objective of providing efficient
 storage and retrieval of timeseries-oriented data and resource
 representations.

 The goal is to experiment with a potential approach to addressing
 an architectural misstep made in the very earliest days of
 ceilometer, specifically the decision to store snapshots of some
 resource metadata alongside each metric datapoint. The core idea
 is to move to storing datapoints shorn of metadata, and instead
 allow the resource-state timeline to be reconstructed more cheaply
 from much less frequently occurring events (e.g. instance resizes
 or migrations).


 What gnocchi isn't:
 ==

 Gnocchi is not a large-scale under-the-radar rewrite of a core
 OpenStack component along the lines of keystone-lite.

 The change is concentrated on the final data-storage phase of
 the ceilometer pipeline, so will have little initial impact on the
 data-acquiring agents, or on transformation phase.

 We've been totally open at the Atlanta summit and other forums
 about this approach being a multi-cycle effort.


 Why we decided to do it this way:
 

 The intent behind spinning up a separate project on stackforge
 was to allow the work progress at arms-length from ceilometer,
 allowing normalcy to be maintained on the core project and a
 rapid rate of innovation on gnocchi.

 Note that that the developers primarily contributing to gnocchi
 represent a cross-section of the core team, and there's a regu

Re: [openstack-dev] [qa][ceilometer] swapping the roles of mongodb and sqlalchemy for ceilometer in Tempest

[Joe Gordon]

On Sat, Aug 9, 2014 at 6:29 AM, Eoghan Glynn  wrote:

>
>
> > > Hi Folks,
> > >
> > > Dina Belova has recently landed some infra patches[1,2] to create
> > > an experimental mongodb-based Tempest job. This effectively just
> > > overrides the ceilometer storage backend config so that mongodb
> > > is used instead of sql-alchemy. The new job has been running
> > > happily for a few days so I'd like now to consider the path
> > > forwards with this.
> > >
> > > One of our Juno goals under the TC gap analysis was to more fully
> > > gate against mongodb, given that this is the storage backend
> > > recommended/supported by many distros. The sql-alchemy backend,
> > > on the other hand, is more suited for proofs of concept or small
> > > deployments. However up to now we've been hampered from reflecting
> > > that reality in the gate, due to the gate being stuck on Precise
> > > for a long time, as befits LTS, and the version of mongodb needed
> > > by ceilometer (i.e. 2.4) effectively unavailable on that Ubuntu
> > > release (in fact it was limited to 2.0.4).
> > >
> > > So the orientation towards gating on sql-alchemy was mostly
> > > driven by legacy issues in the gate's usage of Precise, as
> > > opposed to this being considered the most logical basket in
> > > which to put all our testing eggs.
> > >
> > > However, we're now finally in the brave new world of Trusty :)
> > > So I would like to make the long-delayed change over soon.
> > >
> > > This would involve transposing the roles of sql-alchemy and
> > > mongodb in the gate - the mongodb variant becomes the "blessed"
> > > job run by default, whereas the sql-alchemy based job to
> > > relegated to the second tier.
> > >
> > > So my questions are:
> > >
> > > (a) would the QA side of the house be agreeable to this switch?
> > >
> > > and:
> > >
> > > (b) how long would the mongodb job need to be stable in this
> > > experimental mode before we pull the trigger on swicthing?
> > >
> > > If the answer to (a) is yes, we can get infra patches proposed
> > > early next week to make the swap.
> > >
> > > Cheers,
> > > Eoghan
> > >
> > > [1]
> > >
> https://review.openstack.org/#/q/project:openstack-infra/config+branch:master+topic:ceilometer-mongodb-job,n,z
> > > [2]
> > >
> https://review.openstack.org/#/q/project:openstack-infra/devstack-gate+branch:master+topic:ceilometer-backend,n,z
> > >
> >
> > My interpretation of the gap analysis [1] is merely that you have
> coverage,
> > not that you switch to it and relegate the SQLAlchemy tests to second
> chair.
> > I believe that's a dangerous departure from current standards. A
> dependency
> > on mongodb, due to it's AGPL license, and lack of sufficient support for
> a
> > non-AGPL storage back end, has consistently been raised as a blocking
> issue
> > for Marconi. [2]
>
> Sure, the main goal is to have full mongodb-based coverage in the gate.
>
> So, if the QA/infra folks are prepared to host *both* jobs, then I'd be
> happy to change my request to simply:
>
>   let's promote the mongodb-based Tempest variant to the first tier,
>   to run alongside the current sqlalchemy-based job
>


Ignoring the question of is it ok to say: 'to run ceilometer in any sort of
non-trivial deployment you must manager yet another underlying service,
mongodb' I would prefer not adding an addition gate variant to all
projects.  With the effort to reduce the number of gate variants we have
[0] I would prefer to see just ceilometer gate on both mongodb and
sqlalchemy and the main integrated gate [1] pick just one.

[0] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html
[1]
http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n238


>
> Does that work for you Devananda?
>
> Cheers,
> Eoghan
>
> > -Deva
> >
> >
> > [1]
> >
> https://wiki.openstack.org/wiki/Governance/TechnicalCommittee/Ceilometer_Gap_Coverage
> >
> > [2]
> http://lists.openstack.org/pipermail/openstack-dev/2014-March/030510.html
> > is a very articulate example of this objection
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev