Re: [openstack-dev] [release][requirements] disable constraint bot updates for our own libraries

[Alec Hothan (ahothan)]

From: Jeremy Stanley 
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 

Date: Tuesday, January 17, 2017 at 10:15 AM
To: "OpenStack Development Mailing List (not for usage questions)" 

Subject: Re: [openstack-dev] [release][requirements] disable constraint bot 
updates for our own libraries

On 2017-01-17 18:48:59 +0100 (+0100), Dmitry Tantsur wrote:
[...]
In theory there is nothing wrong with this, as 1.10 is the latest
release indeed. In practice, that means pulling in something with
stable/newton requirements into master, which is concerning, I
agree.
[...]

I don't really see why this is a problem at all. The change in
question updated master constraints from 1.9.0 (a pre-Newton
release) to 1.10.0 (a stable Newton release). Did anything
substantial change in stable/newton between 1.9.0 and 1.10.0 to make
the newer version unsuitable for use with master branch versions of
other projects? Newer is newer is newer. If projects need
integration testing against the master branch (or any particular
branch) of something, they need to be installing from source and not
packages. If the package corresponding to this tag from the stable
branch works with master versions of other projects, then it seems
like our automation worked as intended. Is there a reason to think
that our master branches should be using _older_ versions of
dependencies than our stable branches?


Ksjdlkjads







Granted, it's unclear to me why a stable branch got a release tagged
with a version which semver says is more than straight up bug fixes.
That would seem to fly in the face of stable branch change policy
(but is orthogonal to the topic of this thread).
--
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

[Alec Hothan (ahothan)]

This topic of oslo messaging issues has been going on for a long time and the 
main issue is not the transport itself (each transport has its own limitations) 
but the code using oslo messaging (e.g. pieces of almost every openstack 
service). It is relatively easy to write code using oslo messaging that works 
with devstack or a small scale deployment, it is much less easy to write such 
code that works under the conditions of operations at scale: frequent lack of 
an appropriate test platform, limitations in existing testing tools and to top 
it all, "fuzzy" oslo messaging API definition makes the handling of abnormal 
conditions and load conditions very unpredictable and inconsistent across 
components.

You can't solve this by just "fixing" the oslo messaging layer or by swapping 
to another transport (you'll just open up another can of worms)

As suggested by Ian below, the only practical way to fix this is to define a 
new set of APIs that is much more strictly defined, have openstack code migrate 
to these new APIs and test adequately.
That is clearly very difficult to do with resources moving away from "stable 
and mature" services and attracted by the latest buzzwords (such as containers).

On the original topic of this thread, having geographical distribution will 
certainly introduce a new set of issues at scale.


  Alec

 






On 9/1/16, 6:52 AM, "Ken Giusti"  wrote:

>On Wed, Aug 31, 2016 at 3:30 PM, Ian Wells  wrote:
>> On 31 August 2016 at 10:12, Clint Byrum  wrote:
>>>
>>> Excerpts from Duncan Thomas's message of 2016-08-31 12:42:23 +0300:
>>> > On 31 August 2016 at 11:57, Bogdan Dobrelya 
>>> > wrote:
>>> >
>>> > > I agree that RPC design pattern, as it is implemented now, is a major
>>> > > blocker for OpenStack in general. It requires a major redesign,
>>> > > including handling of corner cases, on both sides, *especially* RPC
>>> > > call
>>> > > clients. Or may be it just have to be abandoned to be replaced by a
>>> > > more
>>> > > cloud friendly pattern.
>>> >
>>> >
>>> > Is there a writeup anywhere on what these issues are? I've heard this
>>> > sentiment expressed multiple times now, but without a writeup of the
>>> > issues
>>> > and the design goals of the replacement, we're unlikely to make progress
>>> > on
>>> > a replacement - even if somebody takes the heroic approach and writes a
>>> > full replacement themselves, the odds of getting community by-in are
>>> > very
>>> > low.
>>>
>>> Right, this is exactly the sort of thing I'd like to gather a group of
>>> design-minded folks around in an Architecture WG. Oslo is busy with the
>>> implementations we have now, but I'm sure many oslo contributors would
>>> like to come up for air and talk about the design issues, and come up
>>> with a current design, and some revisions to it, or a whole new one,
>>> that can be used to put these summit hallway rumors to rest.
>>
>>
>> I'd say the issue is comparatively easy to describe.  In a call sequence:
>>
>> 1. A sends a message to B
>> 2. B receives messages
>> 3. B acts upon message
>> 4. B responds to message
>> 5. A receives response
>> 6. A acts upon response
>>
>> ... you can have a fault at any point in that message flow (consider crashes
>> or program restarts).  If you ask for something to happen, you wait for a
>> reply, and you don't get one, what does it mean?  The operation may have
>> happened, with or without success, or it may not have gotten to the far end.
>> If you send the message, does that mean you'd like it to cause an action
>> tomorrow?  A year from now?  Or perhaps you'd like it to just not happen?
>> Do you understand what Oslo promises you here, and do you think every person
>> who ever wrote an RPC call in the whole OpenStack solution also understood
>> it?
>>
>
>Precisely - IMHO it's a shortcoming of the current o.m. RPC (and
>Notification) API in that it does not let the API user explicitly set
>the desired delivery guarantee when publishing.  Right now it's
>implied that the delivery guarantee is "At Most Once" but that's
>mostly not precisely defined in any meaningful way.
>
>Any messaging API should be explicit regarding what delivery
>guarantee(s) are possible.  In addition, an API should allow the user
>to designate the importance of a message on a per-send basis:  can
>this message be dropped?  can this message be duplicated?  At what
>point in time does the message become invalid (already offered for RPC
>via timeout, but not Notifications IIRC), etc
>
>And well-understood failure modes... things always fail...
>
>
>> I have opinions about other patterns we could use, but I don't want to push
>> my solutions here, I want to see if this is really as much of a problem as
>> it looks and if people concur with my summary above.  However, the right
>> approach is most definitely to create a new and more fitting set of oslo
>> interfaces for communication 

Re: [openstack-dev] [kloudbuster] test LBAAS at scale

[Alec Hothan (ahothan)]

Hi Akshay,

I suppose you're talking about LBAAS v2?

Adding support for lbaas in kloudbuster will require some amount of work which 
can be kept to a minimum if done properly, this addition would be a pretty good 
way to test lbaas at scale.
The tricky part is to modify the staging code without breaking the other 
features (multicast and storage) since this staging is specific to HTTP scale 
test.
The current staging for HTTP scale is based on the following template (I show 
the server side only):

[Router-[HTTP server VM]*]*

The natural extension for supporting LBAAS is to replace each HTTP server with 
a LB group + N HTTP servers:

[Router--[LB---[HTTP server VM]*]*]*

Implementing this would require the following modifications (just a rough 
description of the tasks):

  *   add an additional config option to specify the number of server VMs per 
LB group (defaults to none/no LB) 
  *   if LB is configured, the current config server count would become a LB 
group count
  *   the staging code for the HTTP servers needs to be modified to handle the 
case of LB: 
 *   instead of creating as many HTTP servers as the server count argument, 
create as many LB groups
 *   for each LB group, create the requested HTTP server VMs per group and 
add them to the group
  *   floating IP if requested need to apply to the LB port instead of the HTTP 
servers 
  *   naturally the teardown code will have to also support cleaning up LB 
resources 

  *   HTTP clients will need to connect to the LB VIP address (instead of the 
HTTP server IP address) 

I can help you go through these individual tasks in detail in the code if you 
feel you can handle that, it's just python coding.


The VMs running the HTTP traffic generators are currently always associated 1:1 
to a server VM. With the above template extension you will end up with as many 
HTTP client VMs as LB groups:

(removed the router for better clarity):

[HTTP client VM---[LB---[HTTP server VM]*]*]*

This is not ideal because each HTTP traffic generator can only support a 
relatively low number of connections (in the few thousands) while an HTTP 
server instance can easily support many times this load especially for light 
HTTP traffic (i.e. replies that are very short).

So another improvement (which we had on our roadmap) would be to support N:1 
mapping:

[[HTTP client VM]*LB---[HTTP server VM]*]*]*

this could be a separate extension.
Let me know if you'd like to do this and we can help navigate the code.

Thanks

   Alec



From: Akshay Kumar Sanghai 
>
Date: Tuesday, August 23, 2016 at 2:07 PM
To: Alec Hothan >
Cc: "Yichen Wang (yicwang)" >, 
"OpenStack Development Mailing List (not for usage questions)" 
>
Subject: Re: [openstack-dev] [kloudbuster] authorization failed problem

Hi Yichen, Alec,

The kloudbuster project worked perfectly fine for me. Now I want to integrate 
lbaas for scale testing. Can you guys help in how do i achieve that? Please 
include me for any contribution.

Thanks
Akshay Sanghai


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kloudbuster] authorization failed problem

[Alec Hothan (ahothan)]

From: Akshay Kumar Sanghai 
<akshaykumarsang...@gmail.com<mailto:akshaykumarsang...@gmail.com>>
Date: Thursday, April 7, 2016 at 1:46 AM
To: "Yichen Wang (yicwang)" <yicw...@cisco.com<mailto:yicw...@cisco.com>>, Alec 
Hothan <ahot...@cisco.com<mailto:ahot...@cisco.com>>
Cc: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [kloudbuster] authorization failed problem

Thanks Yichen and Alec.
Yichen, It was what I was looking for. I started with Rally , but faced problem 
with defining the number of router per tenant and traffic generation. Then, I 
found that there is a new project Kloudbuster. I faced some issues , but with 
your help , I succeeded in running it.
One suggestion: I think Rally developers  are also trying for the traffic 
generation. So just make sure, we don't have two things for the same work under 
the openstack umbrella.

We're also using Rally to test our control plane. Data plane testing at scale 
is a whole different ballgame (than control plane) and we had very specific 
scale test requirements in mind for our own usage as far back as late 2014, 
which is why we developed kloudbuster in Feb 2015 with HTTP traffic at scale. 
At that time there was no solution available for doing automated data plane 
scale testing, even today as you can see the number of solutions is actually 
very limited.
Anyway if the Rally developers are interested to discuss about it, I or Yichen 
would be happy to discuss.


One Request:  I have not contributed to any project till now. I am interested 
in contributing to the Kloudbuster project. Please suggest how to start and 
help me in getting up to speed.

There is currently no bug backlog in kloudbuster. Since we use it quite a lot, 
every issue we run into is fixed pretty quickly but it is likely there are 
uncovered issues when run in a different environment.
There are also obviously many feature enhancements of any size but we only add 
when there is an ask as we want to keep the code small (do one thing but do it 
well).
If you use it and find a bug or need some enhancement, you're welcome to 
contribute and we can help root cause and fix.
Have you been looking for traffic generation for any particular purpose? We use 
it for testing our openstack data plane at scale (and storage now).

Regards,

  Alec




Regards,
Akshay

On Wed, Apr 6, 2016 at 1:44 PM, Yichen Wang (yicwang) 
<yicw...@cisco.com<mailto:yicw...@cisco.com>> wrote:
Hi, Akshay,

Just curious, how do you find KloudBuster so far? Does it do its job and fit 
your needs? ☺

Thanks very much!

Regards,
Yichen

From: Alec Hothan (ahothan)
Sent: Tuesday, April 5, 2016 9:54 AM
To: Akshay Kumar Sanghai 
<akshaykumarsang...@gmail.com<mailto:akshaykumarsang...@gmail.com>>; Yichen 
Wang (yicwang) <yicw...@cisco.com<mailto:yicw...@cisco.com>>
Cc: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>

Subject: Re: [openstack-dev] [kloudbuster] authorization failed problem


Akshay,

Note that version 6 is now released so please use the official image from the 
OpenStack App Catalog and update your code to latest.
The doc has also been updated, you might want to have a look at the new arch 
section and gallery - those should help you with the questions you had below 
regarding the scale test staging and traffic flows.
http://kloudbuster.readthedocs.org/en/latest/index.html

Thanks

   Alec


From: Akshay Kumar Sanghai 
<akshaykumarsang...@gmail.com<mailto:akshaykumarsang...@gmail.com>>
Date: Wednesday, March 30, 2016 at 11:11 AM
To: "Yichen Wang (yicwang)" <yicw...@cisco.com<mailto:yicw...@cisco.com>>
Cc: Alec Hothan <ahot...@cisco.com<mailto:ahot...@cisco.com>>, OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [kloudbuster] authorization failed problem

Hi Yichen,
Thanks a lot . I will try with v6 and reach out to you for further help.

Regards,
Akshay

On Wed, Mar 30, 2016 at 11:35 PM, Yichen Wang (yicwang) 
<yicw...@cisco.com<mailto:yicw...@cisco.com>> wrote:
Hi, Akshay,

From the log you attached, the good news is you got KloudBuster installed and 
running fine! The problem is the image you are using (v5) is outdated for the 
latest KloudBuster main code. ☺

Normally for every version of KloudBuster, it needs certain version of image to 
support the full functionality. In the case when new feature is brought in, we 
tag the main code with a new version, and bump up the image version. Like from 
v5 to v6, we added the capability to support storage testing on cinder volume 
and ephemeral disks as well. We are right in our time for publishing the v6 
image to the OpenStack App Catalog, 

Re: [openstack-dev] [kloudbuster] authorization failed problem

[Alec Hothan (ahothan)]

Akshay,

Note that version 6 is now released so please use the official image from the 
OpenStack App Catalog and update your code to latest.
The doc has also been updated, you might want to have a look at the new arch 
section and gallery - those should help you with the questions you had below 
regarding the scale test staging and traffic flows.
http://kloudbuster.readthedocs.org/en/latest/index.html

Thanks

   Alec


From: Akshay Kumar Sanghai 
<akshaykumarsang...@gmail.com<mailto:akshaykumarsang...@gmail.com>>
Date: Wednesday, March 30, 2016 at 11:11 AM
To: "Yichen Wang (yicwang)" <yicw...@cisco.com<mailto:yicw...@cisco.com>>
Cc: Alec Hothan <ahot...@cisco.com<mailto:ahot...@cisco.com>>, OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [kloudbuster] authorization failed problem

Hi Yichen,
Thanks a lot . I will try with v6 and reach out to you for further help.

Regards,
Akshay

On Wed, Mar 30, 2016 at 11:35 PM, Yichen Wang (yicwang) 
<yicw...@cisco.com<mailto:yicw...@cisco.com>> wrote:
Hi, Akshay,

From the log you attached, the good news is you got KloudBuster installed and 
running fine! The problem is the image you are using (v5) is outdated for the 
latest KloudBuster main code. ☺

Normally for every version of KloudBuster, it needs certain version of image to 
support the full functionality. In the case when new feature is brought in, we 
tag the main code with a new version, and bump up the image version. Like from 
v5 to v6, we added the capability to support storage testing on cinder volume 
and ephemeral disks as well. We are right in our time for publishing the v6 
image to the OpenStack App Catalog, which may take another day or two. This is 
why you are seeing the connection to the redis agent in KB-Proxy is failing…

In order to unblock you, here is the RC image of v6 we are using right now, 
replace it in your cloud and KloudBuster should be good to go:
https://cisco.box.com/s/xelzx15swjra5qr0ieafyxnbyucnnsa0

Now back to your question.
-Does the server side means the cloud generating the traffic and client side 
means the the cloud on which connections are established? Can you please 
elaborate on client, server and proxy?
[Yichen] It is the other way around. Server is running nginx, and client is 
running the traffic generator (wrk2). It is like the way we normally 
understand. Since there might be lots of servers and clients in the same cloud, 
so KB-Proxy is an additional VM that runs in the clients side to orchestrate 
all client VMs to generate traffic, collect the results from each VM, and send 
them back to the main KloudBuster for processing. KB-Proxy is the where the 
redis server is sitting, and acts as the proxy node to connect all internal VMs 
to the external network. This is why a floating IP is needed for the proxy node.

-while running the kloudbuster, I saw "setting up redis connection". Can you 
please expain to which connection is established and why? Is it KB_PROXY?
[Yichen] As I explained above, KB-Proxy is the bridge between internal VM and 
external world (like the host you are running KloudBuster from). “Setting up 
redis connection” means the KloudBuster is trying to connect to the redis 
server on the KB-Proxy node. You may see some retries because it does take some 
time for the VM to be up running.

Thanks very much!

Regards,
Yichen

From: Akshay Kumar Sanghai 
[mailto:akshaykumarsang...@gmail.com<mailto:akshaykumarsang...@gmail.com>]
Sent: Wednesday, March 30, 2016 7:31 AM
To: Alec Hothan (ahothan) <ahot...@cisco.com<mailto:ahot...@cisco.com>>
Cc: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>; 
Yichen Wang (yicwang) <yicw...@cisco.com<mailto:yicw...@cisco.com>>

Subject: Re: [openstack-dev] [kloudbuster] authorization failed problem

Hi Alec,
Thanks for clarifying. I didnot have the cinder service previously. It was not 
a complete setup. Now, I did the setup of cinder service.
Output of keystone service list.
[Inline image 1]
I installed the setup of openstack using the installation guide for ubuntu and 
for kloudbuster, its a pypi based installation. So, I am running kloudbuster 
using the CLI option.
kloudbuster --tested-rc keystone-openrc.sh --tested-passwd * --config kb.cfg

contents of kb.cfg:
image_name: 'kloudbuster'

I added the kloudbuster v5 version as glance image with name as kloudbuster.

I don't understand some basic things. If you can help, then that would be great.
-Does the server side means the cloud generating the traffic and client side 
means the the cloud on which connections are established? Can you please 
elaborate on client, server and proxy?
-while running the kloudbuster, I saw "setting up redis connection". Can you 
please expain to which connection is established and

Re: [openstack-dev] [kloudbuster] authorization failed problem

[Alec Hothan (ahothan)]

Can you describe what you mean by "do not have a cinder service"?
Can you provide the output of "keystone service-list"?

We'd have to know a bit more about what you have been doing:
how did you install your openstack, how did you install kloudbuster, which 
kloudbuster qcow2 image version did you use, who did you run kloudbuster (cli 
or REST or web UI), what config file have you been using, complete log of the 
run (including backtrace)...

But the key is - you should really have a fully working openstack deployment 
before using kloudbuster. Nobody has never tried so far to use kloudbuster 
without such basic service as cinder working.

Thanks

  Alec



From: Akshay Kumar Sanghai 
<akshaykumarsang...@gmail.com<mailto:akshaykumarsang...@gmail.com>>
Date: Monday, March 28, 2016 at 6:51 AM
To: OpenStack List 
<openstack-dev@lists.openstack.org<mailto:openstack-dev@lists.openstack.org>>, 
Alec Hothan <ahot...@cisco.com<mailto:ahot...@cisco.com>>
Cc: "Yichen Wang (yicwang)" <yicw...@cisco.com<mailto:yicw...@cisco.com>>
Subject: Re: [openstack-dev] [kloudbuster] authorization failed problem

Hi Alec,
Thanks for the help. I ran into another problem. At present I do not have a 
cinder service. So ,when i am trying to run kloudbuster, I am getting this 
error:
"EndpointNotFound: publicURL endpoint for volumev2 service not found"
Is it possible to run the scale test (creation of VMs, router, network) without 
having a cinder service? Any option that can be used so that kloudbuster can 
run without cinder.

Thanks,
Akshay

On Wed, Mar 23, 2016 at 9:05 PM, Alec Hothan (ahothan) 
<ahot...@cisco.com<mailto:ahot...@cisco.com>> wrote:
Hi Akshay

The URL you are using is a private address (http://192.168.138.51:5000/v2.0) 
and is likely the reason it does not work.
If you run the kloudbuster App in the cloud, this app needs to have access to 
the cloud under test.
So even if you can access 192.168.138.51 from your local browser (which runs on 
your workstation or laptop) it may not be accessible from a VM that runs in 
your cloud.
For that to work you need to get an URL that is reachable from the VM.

In some cases where the cloud under test is local, it is easier to just run 
kloudbuster locally as well (from the same place where you can ping 
192.168.138.51).
You can either use a local VM to run the kloudbuster image (vagrant, virtual 
box...) or just simpler, install kloudbuster locally using git clone or pip 
install (see the installation instructions in the doc 
http://kloudbuster.readthedocs.org/en/latest/).

Regards,

   Alec


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kloudbuster] authorization failed problem

[Alec Hothan (ahothan)]

Hi Akshay

The URL you are using is a private address (http://192.168.138.51:5000/v2.0) 
and is likely the reason it does not work.
If you run the kloudbuster App in the cloud, this app needs to have access to 
the cloud under test.
So even if you can access 192.168.138.51 from your local browser (which runs on 
your workstation or laptop) it may not be accessible from a VM that runs in 
your cloud.
For that to work you need to get an URL that is reachable from the VM.

In some cases where the cloud under test is local, it is easier to just run 
kloudbuster locally as well (from the same place where you can ping 
192.168.138.51).
You can either use a local VM to run the kloudbuster image (vagrant, virtual 
box...) or just simpler, install kloudbuster locally using git clone or pip 
install (see the installation instructions in the doc 
http://kloudbuster.readthedocs.org/en/latest/).

Regards,

   Alec




From: Akshay Kumar Sanghai 
>
Reply-To: 
"openstack-dev@lists.openstack.org" 
>
Date: Wednesday, March 23, 2016 at 6:59 AM
To: "Yichen Wang (yicwang)" >, 
"openstack-dev@lists.openstack.org" 
>
Subject: [openstack-dev] [kloudbuster] authorization failed problem


Hi,

I am trying to use cloudbuster for the scale testing of openstack setup.

I have a openstack setup with 1 controller, 1 network and 2 compute node. I am 
trying to use to use kloudbuster for scale testing of the setup. I created one 
VM with kloudbuster image. I accessed the web UI and clicked on "stage". This 
is the log:
:23,206 WARNING No public key is found or specified to instantiate VMs. You 
will not be able to access the VMs spawned by KloudBuster.
2016-03-22 14:01:30,464 WARNING Traceback (most recent call last):
  File \"/kb_test/kloudbuster/kb_server/kb_server/controllers/api_kb.py\", line 
58, in kb_stage_thread_handler
if kb_session.kloudbuster.check_and_upload_images():
  File 
\"/kb_test/kloudbuster/kb_server/kb_server/controllers/../../../kloudbuster/kloudbuster.py\",
 line 283, in check_and_upload_images
keystone_list = [create_keystone_client(self.server_cred)[0],
  File 
\"/kb_test/kloudbuster/kb_server/kb_server/controllers/../../../kloudbuster/kloudbuster.py\",
 line 54, in create_keystone_client
return (keystoneclient.Client(endpoint_type='publicURL', **creds), 
creds['auth_url'])
  File 
\"/usr/local/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py\", line 
166, in __init__
self.authenticate()
  File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py\", line 
337, in inner
return func(*args, **kwargs)
  File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/httpclient.py\", 
line 589, in authenticate
resp = self.get_raw_token_from_identity_service(**kwargs)
  File 
\"/usr/local/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py\", line 
210, in get_raw_token_from_identity_service
_(\"Authorization Failed: %s\") % e)
AuthorizationFailure: Authorization Failed: Unable to establish connection to 
http://192.168.138.51:5000/v2.0/tokens

I used a rest client to find whether v2.0/tokens is working or not, it was 
working. I got the token . This is the openrc file I used:
export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
#export OS_IMAGE_API_VERSION=2
export OS_AUTH_URL=http://192.168.138.51:5000/v2.0
export OS_TENANT_NAME=admin
#export OS_PROJECT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=sanghai
export OS_REGION_NAME=RegionOne

Please suggest a solution and let me know if I missed some details.

Thanks,
Akshay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Scheduler proposal

[Alec Hothan (ahothan)]

On 10/15/15, 11:11 PM, "Clint Byrum"  wrote:

>Excerpts from Ed Leafe's message of 2015-10-15 11:56:24 -0700:
>> Wow, I seem to have unleashed a bunch of pent-up frustration in the 
>> community! It's great to see everyone coming forward with their ideas and 
>> insights for improving the way Nova (and, by extension, all of OpenStack) 
>> can potentially scale.
>> 
>> I do have a few comments on the discussion:
>> 
>> 1) This isn't a proposal to simply add some sort of DLM to Nova as a magic 
>> cure-all. The concerns about Nova's ability to scale have to do a lot more 
>> with the overall internal communication design.
>> 
>
>In this, we agree.
>
>> 2) I really liked the comment about "made-up numbers". It's so true: we are 
>> all impressed by such examples of speed that we sometimes forget whether 
>> speeding up X will improve the overall process to any significant degree. 
>> The purpose of my original email back in July, and the question I asked at 
>> the Nova midcycle, is if we could get some numbers that would be a target to 
>> shoot for with any of these experiments. Sure, I could come up with a test 
>> that shows a zillion transactions per second, but if that doesn't result in 
>> a cloud being able to schedule more efficiently, what's the point?
>>
>
>Speed is only 1 dimension. Efficiency and simplicity are two others that
>I think are harder to quantify, but are also equally important in any
>component of OpenStack.

Monty did suggest a goal with 100K nodes - which I think is moon expedition 
kind of goal given how far we are from it, but goal nevertheless ;-)
Openstack does not provide any number today outside of "massive scale" and this 
could be a problem for designers and implementors.
I think openstack is now mature enough to have to worry seriously about scale, 
and we have a very long way to go at that level.

I agree for the importance of simplicity and efficiency. But lets also add 
operational requirements such as ease of deployment and ease of 
troubleshooting. It is more difficult for Ops to deal with too many different 
technologies under the cover.
My concern is we may not have sufficient oversight (from the TC) for this kind 
of project to keep it within reasonable complexity for the given requirements, 
and this is hard to achieve when the requirements are very vague.
It looks like the main area where we might need faster nova scheduling would be 
those big deployments that use nova networking (thousands of nodes) - just 
because neutron deployments just may not have enough nodes to require such 
rate. And nobody seems to know what is the targeted rate (schedules per second 
here) and what is the exact problem to solve (by exact I mean what numbers do 
we have to say that the current nova scheduling is too slow or does not scale).


>
>> 3) I like the idea of something like ZooKeeper, but my concern is how to 
>> efficiently query the data. If, for example, we had records for 100K compute 
>> nodes, would it be possible to do the equivalent of "SELECT * FROM resources 
>> WHERE resource_type = 'compute' AND free_ram_mb >= 2048 AND …" - well, you 
>> get the idea. Are complex data queries possible in ZK? I haven't been able 
>> to find that information anywhere.
>>
>
>You don't do complex queries, because you have all of the data in RAM,
>in an efficient in-RAM format. Even if each record is 50KB, we can do
>100,000 of them in 5GB. That's a drop in the bucket.

Yes

>
>> 4) It is true that even in a very large deployment, it is possible to keep 
>> all the relevant data needed for scheduling in memory. My concern is how to 
>> efficiently search that data, much like in the ZK scenario.
>> 
>
>There are a bunch of ways to do this. My favorite is to have filter
>plugins in the scheduler define what they need to index, and then
>build a B-tree for each filter as each record arrives in the main data
>structure. When scheduling requests come in, they simply walk through
>each B-tree and turn that into a set. Then read each piece of the set
>out of the main structure and sort based on whichever you want (less
>full for load balancing, most full for efficient stacking).

There are clearly things you should be doing to scale properly. Python is not 
very speedy but can be made good enough at scale using the proper algorithm 
(such as the one you propose).
Furthermore, it can be made to run much faster - close to native speed - with 
proper design and the use of the proper libraries. So there is a lot that can 
be done to speed up things without having necessarily to increase complexity 
and scale out everything.


>
>> 5) Concerns about Cassandra running with OpenJDK instead of the Oracle JVM 
>> are troubling. I sent an email about this to one of the people I know at 
>> DataStax, but so far have not received a response. And while it would be 
>> great to have people contribute to OpenJDK to make it compatible, keep in 
>> mind that that would be an ongoing 

Re: [openstack-dev] Scheduler proposal

[Alec Hothan (ahothan)]

On 10/12/15, 12:05 PM, "Monty Taylor" <mord...@inaugust.com> wrote:

>On 10/12/2015 02:45 PM, Joshua Harlow wrote:
>> Alec Hothan (ahothan) wrote:
>>>
>>>
>>>
>>>
>
>I want to do 100k hypervisors. No, that's not hyperbole.
>
>Also, I do not think that ZK/consul/etcd are very costly for small 
>deployments. Given the number of simple dev-oriented projects that start 
>with "so install ZK/consul/etcd" I think they've all proven their 
>ability to scale _down_ - and I'm also pretty sure all of them have 
>installations that clear 100k nodes.
>
>This:
>
>to produce the ubiquitous Open Source Cloud Computing platform that will 
>meet the needs of public and private clouds regardless of size, by being 
>simple to implement and massively scalable.
>
>is what we're doing.
>
>Our mission is NOT "produce a mid-range cloud that is too complex for 
>small deployments and tops out before you get to big ones"
>
>I don't think "handle massive clouds" has ever NOT been on the list of 
>stated goals. (that mission statement has not changed since we started 
>the project - although I agree with Joe, it's in need of an update- 
>there is no mention of users)

Then it'd be great that there be an official statement from the TC about the 
scale objectives and if possible put some numbers, "massive cloud" is ambiguous 
for folks who actually have to make sure they scale to specs.
So should mention "OpenStack should scale from 1 node to 100K nodes" for 
example. As long as everybody is fully aware about how far we are today from 
that lofty goal.
This clearly will have an impact on how we need to design services and how we 
should change the way we test for them. It will be tricky to get a 1000 node 
lab up and running just for openstack developers, it is just not practical at 
all. The only practical way will be to do proper unit testing at scale (e.g. 
emulate a 10K node cloud for unit testing any given service).


>
>BTW - Infra runs against currently runs against clouds rate-limited at 
>roughly 10 api calls / second. That's just one tenant - but it's a 
>perfectly managable rate. Now, if the cloud could continue to add nodes 
>and users without that rate degrading I think we'd be in really good shape.

I think that rate limit only applies to REST APIs, I don't think there is any 
rate limit for oslo messaging.
Even only 10 API calls per second per tenant can be a challenge with a large 
number of tenants. I don't think there is any provision today for example to 
ensure fairness across tenants.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Scheduler proposal

[Alec Hothan (ahothan)]

On 10/10/15, 11:35 PM, "Clint Byrum" <cl...@fewbar.com> wrote:

>Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700:
>> 
>> On 10/9/15, 6:29 PM, "Clint Byrum" <cl...@fewbar.com> wrote:
>> 
>> >Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700:
>> >> On 10/09/2015 03:36 PM, Ian Wells wrote:
>> >> > On 9 October 2015 at 12:50, Chris Friesen <chris.frie...@windriver.com
>> >> > <mailto:chris.frie...@windriver.com>> wrote:
>> >> >
>> >> > Has anybody looked at why 1 instance is too slow and what it would 
>> >> > take to
>> >> >
>> >> > make 1 scheduler instance work fast enough? This does not 
>> >> > preclude the
>> >> > use of
>> >> > concurrency for finer grain tasks in the background.
>> >> >
>> >> >
>> >> > Currently we pull data on all (!) of the compute nodes out of the 
>> >> > database
>> >> > via a series of RPC calls, then evaluate the various filters in 
>> >> > python code.
>> >> >
>> >> >
>> >> > I'll say again: the database seems to me to be the problem here.  Not to
>> >> > mention, you've just explained that they are in practice holding all 
>> >> > the data in
>> >> > memory in order to do the work so the benefit we're getting here is 
>> >> > really a
>> >> > N-to-1-to-M pattern with a DB in the middle (the store-to-DB is rather
>> >> > secondary, in fact), and that without incremental updates to the 
>> >> > receivers.
>> >> 
>> >> I don't see any reason why you couldn't have an in-memory scheduler.
>> >> 
>> >> Currently the database serves as the persistant storage for the resource 
>> >> usage, 
>> >> so if we take it out of the picture I imagine you'd want to have some way 
>> >> of 
>> >> querying the compute nodes for their current state when the scheduler 
>> >> first 
>> >> starts up.
>> >> 
>> >> I think the current code uses the fact that objects are remotable via the 
>> >> conductor, so changing that to do explicit posts to a known scheduler 
>> >> topic 
>> >> would take some work.
>> >> 
>> >
>> >Funny enough, I think thats exactly what Josh's "just use Zookeeper"
>> >message is about. Except in memory, it is "in an observable storage
>> >location".
>> >
>> >Instead of having the scheduler do all of the compute node inspection
>> >and querying though, you have the nodes push their stats into something
>> >like Zookeeper or consul, and then have schedulers watch those stats
>> >for changes to keep their in-memory version of the data up to date. So
>> >when you bring a new one online, you don't have to query all the nodes,
>> >you just scrape the data store, which all of these stores (etcd, consul,
>> >ZK) are built to support atomically querying and watching at the same
>> >time, so you can have a reasonable expectation of correctness.
>> >
>> >Even if you figured out how to make the in-memory scheduler crazy fast,
>> >There's still value in concurrency for other reasons. No matter how
>> >fast you make the scheduler, you'll be slave to the response time of
>> >a single scheduling request. If you take 1ms to schedule each node
>> >(including just reading the request and pushing out your scheduling
>> >result!) you will never achieve greater than 1000/s. 1ms is way lower
>> >than it's going to take just to shove a tiny message into RabbitMQ or
>> >even 0mq.
>> 
>> That is not what I have seen, measurements that I did or done by others show 
>> between 5000 and 1 send *per sec* (depending on mirroring, up to 1KB msg 
>> size) using oslo messaging/kombu over rabbitMQ.
>
>You're quoting througput of RabbitMQ, but how many threads were
>involved? An in-memory scheduler that was multi-threaded would need to
>implement synchronization at a fairly granular level to use the same
>in-memory store, and we're right back to the extreme need for efficient
>concurrency in the design, though with much better latency on the
>synchronization.

These were single-threaded tests and you're correct that if you had multiple 
threads trying to send something you'd have some inefficiency.
However I'd q

Re: [openstack-dev] Scheduler proposal

[Alec Hothan (ahothan)]

On 10/12/15, 11:45 AM, "Joshua Harlow" <harlo...@fastmail.com> wrote:

>Alec Hothan (ahothan) wrote:
>>
>>
>>
>>
>> On 10/10/15, 11:35 PM, "Clint Byrum"<cl...@fewbar.com>  wrote:
>>
>>> Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700:
>>>> On 10/9/15, 6:29 PM, "Clint Byrum"<cl...@fewbar.com>  wrote:
>>>>
>>>>> Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700:
>>>>>> On 10/09/2015 03:36 PM, Ian Wells wrote:
>>>>>>> On 9 October 2015 at 12:50, Chris Friesen<chris.frie...@windriver.com
>>>>>>> <mailto:chris.frie...@windriver.com>>  wrote:
>>>>>>>
>>>>>>>  Has anybody looked at why 1 instance is too slow and what it would 
>>>>>>> take to
>>>>>>>
>>>>>>>  make 1 scheduler instance work fast enough? This does not 
>>>>>>> preclude the
>>>>>>>  use of
>>>>>>>  concurrency for finer grain tasks in the background.
>>>>>>>
>>>>>>>
>>>>>>>  Currently we pull data on all (!) of the compute nodes out of the 
>>>>>>> database
>>>>>>>  via a series of RPC calls, then evaluate the various filters in 
>>>>>>> python code.
>>>>>>>
>>>>>>>
>>>>>>> I'll say again: the database seems to me to be the problem here.  Not to
>>>>>>> mention, you've just explained that they are in practice holding all 
>>>>>>> the data in
>>>>>>> memory in order to do the work so the benefit we're getting here is 
>>>>>>> really a
>>>>>>> N-to-1-to-M pattern with a DB in the middle (the store-to-DB is rather
>>>>>>> secondary, in fact), and that without incremental updates to the 
>>>>>>> receivers.
>>>>>> I don't see any reason why you couldn't have an in-memory scheduler.
>>>>>>
>>>>>> Currently the database serves as the persistant storage for the resource 
>>>>>> usage,
>>>>>> so if we take it out of the picture I imagine you'd want to have some 
>>>>>> way of
>>>>>> querying the compute nodes for their current state when the scheduler 
>>>>>> first
>>>>>> starts up.
>>>>>>
>>>>>> I think the current code uses the fact that objects are remotable via the
>>>>>> conductor, so changing that to do explicit posts to a known scheduler 
>>>>>> topic
>>>>>> would take some work.
>>>>>>
>>>>> Funny enough, I think thats exactly what Josh's "just use Zookeeper"
>>>>> message is about. Except in memory, it is "in an observable storage
>>>>> location".
>>>>>
>>>>> Instead of having the scheduler do all of the compute node inspection
>>>>> and querying though, you have the nodes push their stats into something
>>>>> like Zookeeper or consul, and then have schedulers watch those stats
>>>>> for changes to keep their in-memory version of the data up to date. So
>>>>> when you bring a new one online, you don't have to query all the nodes,
>>>>> you just scrape the data store, which all of these stores (etcd, consul,
>>>>> ZK) are built to support atomically querying and watching at the same
>>>>> time, so you can have a reasonable expectation of correctness.
>>>>>
>>>>> Even if you figured out how to make the in-memory scheduler crazy fast,
>>>>> There's still value in concurrency for other reasons. No matter how
>>>>> fast you make the scheduler, you'll be slave to the response time of
>>>>> a single scheduling request. If you take 1ms to schedule each node
>>>>> (including just reading the request and pushing out your scheduling
>>>>> result!) you will never achieve greater than 1000/s. 1ms is way lower
>>>>> than it's going to take just to shove a tiny message into RabbitMQ or
>>>>> even 0mq.
>>>> That is not what I have seen, measurements that I did or done by others 
>>>> show between 5000 and 1 send *per sec* (depending on mirroring, up

Re: [openstack-dev] Scheduler proposal

[Alec Hothan (ahothan)]

On 10/9/15, 6:29 PM, "Clint Byrum"  wrote:

>Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700:
>> On 10/09/2015 03:36 PM, Ian Wells wrote:
>> > On 9 October 2015 at 12:50, Chris Friesen > > > wrote:
>> >
>> > Has anybody looked at why 1 instance is too slow and what it would 
>> > take to
>> >
>> > make 1 scheduler instance work fast enough? This does not preclude 
>> > the
>> > use of
>> > concurrency for finer grain tasks in the background.
>> >
>> >
>> > Currently we pull data on all (!) of the compute nodes out of the 
>> > database
>> > via a series of RPC calls, then evaluate the various filters in python 
>> > code.
>> >
>> >
>> > I'll say again: the database seems to me to be the problem here.  Not to
>> > mention, you've just explained that they are in practice holding all the 
>> > data in
>> > memory in order to do the work so the benefit we're getting here is really 
>> > a
>> > N-to-1-to-M pattern with a DB in the middle (the store-to-DB is rather
>> > secondary, in fact), and that without incremental updates to the receivers.
>> 
>> I don't see any reason why you couldn't have an in-memory scheduler.
>> 
>> Currently the database serves as the persistant storage for the resource 
>> usage, 
>> so if we take it out of the picture I imagine you'd want to have some way of 
>> querying the compute nodes for their current state when the scheduler first 
>> starts up.
>> 
>> I think the current code uses the fact that objects are remotable via the 
>> conductor, so changing that to do explicit posts to a known scheduler topic 
>> would take some work.
>> 
>
>Funny enough, I think thats exactly what Josh's "just use Zookeeper"
>message is about. Except in memory, it is "in an observable storage
>location".
>
>Instead of having the scheduler do all of the compute node inspection
>and querying though, you have the nodes push their stats into something
>like Zookeeper or consul, and then have schedulers watch those stats
>for changes to keep their in-memory version of the data up to date. So
>when you bring a new one online, you don't have to query all the nodes,
>you just scrape the data store, which all of these stores (etcd, consul,
>ZK) are built to support atomically querying and watching at the same
>time, so you can have a reasonable expectation of correctness.
>
>Even if you figured out how to make the in-memory scheduler crazy fast,
>There's still value in concurrency for other reasons. No matter how
>fast you make the scheduler, you'll be slave to the response time of
>a single scheduling request. If you take 1ms to schedule each node
>(including just reading the request and pushing out your scheduling
>result!) you will never achieve greater than 1000/s. 1ms is way lower
>than it's going to take just to shove a tiny message into RabbitMQ or
>even 0mq.

That is not what I have seen, measurements that I did or done by others show 
between 5000 and 1 send *per sec* (depending on mirroring, up to 1KB msg 
size) using oslo messaging/kombu over rabbitMQ.
And this is unmodified/highly unoptimized oslo messaging code.
If you remove the oslo messaging layer, you get 25000 to 45000 msg/sec with 
kombu/rabbitMQ (which shows how inefficient is oslo messaging layer itself)


> So I'm pretty sure this is o-k for small clouds, but would be
>a disaster for a large, busy cloud.

It all depends on how many sched/sec for the "large busy cloud"...

>
>If, however, you can have 20 schedulers that all take 10ms on average,
>and have the occasional lock contention for a resource counter resulting
>in 100ms, now you're at 2000/s minus the lock contention rate. This
>strategy would scale better with the number of compute nodes, since
>more nodes means more distinct locks, so you can scale out the number
>of running servers separate from the number of scheduling requests.

How many compute nodes are we talking about max? How many scheduling per second 
is the requirement? And where are we today with the latest nova scheduler?
My point is that without these numbers we could end up under-shooting, 
over-shooting or over-engineering along with the cost of maintaining that extra 
complexity over the lifetime of openstack.

I'll just make up some numbers for the sake of this discussion:

nova scheduler latest can do only 100 sched/sec for 1 instance (I guess the 
10ms average you bring out may not be that unrealistic)
the requirement is a sustained 500 sched/sec worst case with 10K nodes (that is 
5% of 10K and today we can barely launch 100VM/sec sustained)

Are we going to achieve 5x with just 3 instances which is what most people 
deploy? Not likely.
Will using more elaborate distributed infra/DLM like consul/zk/etcd going to 
get us to that 500 mark with 3 instances? Maybe but it will be at the expense 
of added complexity of the overall solution.
Can we instead optimize 

Re: [openstack-dev] Scheduler proposal

[Alec Hothan (ahothan)]

There are several ways to make python code that deals with a lot of data 
faster, especially when it comes to operating on DB fields from SQL tables (and 
that is not limited to the nova scheduler).
Pulling data from large SQL tables and operating on them through regular python 
code (using python loops) is extremely inefficient due to the nature of the 
python interpreter. If this is what nova scheduler code is doing today, the 
good thing is there is a potentially huge room for improvement.


The approach to scale out, in practice means a few instances (3 instances is 
common), meaning the gain would be in the order of 3x (or 1 order of magnitude) 
but with sharply increased complexity to deal with concurrent schedulers and 
potentially conflicting results (with the use of tools lie ZK or Consul...). 
But in essence we're basically just running the same unoptimized code 
concurrently to achieve a better throughput.
On the other hand optimizing something that is not very optimized to start with 
can yield a much better return than 3x, with the advantage of simplicity (one 
active scheduler, which could be backed by a standby for HA).

Python is actually one of the better languages to do *fast* in-memory big data 
processing using open source python scientific and data analysis libraries as 
they can provide native speed through cythonized libraries and powerful high 
level abstraction to do complex filters and vectorized operations. Not only it 
is fast but it also yields much smaller code.

I have used libraries such as numpy and pandas to operate on very large data 
sets (the equivalent of SQL tables with hundreds of thousands of rows) and 
there is easily 2 orders of magnitude of difference for operating on these data 
in memory between plain python code with loops and python code using these 
libraries (that is without any DB access).
The order of filtering on the kind of reduction that you describe below 
certainly helps but becomes second order when you use pandas filters because 
they are extremely fast even for very large datasets.

I'm curious to know why this path was not explored more before embarking full 
speed on concurrency/scale out options which is a very complex and treacherous 
path as we see in this discussion. Clearly very attractive intellectually to 
work with all these complex distributed frameworks, but the cost of complexity 
is often overlooked.

Is there any data showing the performance of the current nova scheduler? How 
many scheduling can nova do per second at scale with worst case filters?
When you think about it, 10,000 nodes and their associated properties is not 
such a big number if you use the right libraries.




On 10/9/15, 1:10 PM, "Joshua Harlow" <harlo...@fastmail.com> wrote:

>And also we should probably deprecate/not recommend:
>
>http://docs.openstack.org/developer/nova/api/nova.scheduler.filters.json_filter.html#nova.scheduler.filters.json_filter.JsonFilter
>
>That filter IMHO basically disallows optimizations like forming SQL 
>statements for each filter (and then letting the DB do the heavy 
>lifting) or say having each filter say 'oh my logic can be performed by 
>a prepared statement ABC and u should just use that instead' (and then 
>letting the DB do the heavy lifting).
>
>Chris Friesen wrote:
>> On 10/09/2015 12:25 PM, Alec Hothan (ahothan) wrote:
>>>
>>> Still the point from Chris is valid. I guess the main reason openstack is
>>> going with multiple concurrent schedulers is to scale out by
>>> distributing the
>>> load between multiple instances of schedulers because 1 instance is too
>>> slow. This discussion is about coordinating the many instances of
>>> schedulers
>>> in a way that works and this is actually a difficult problem and will get
>>> worst as the number of variables for instance placement increases (for
>>> example NFV is going to require a lot more than just cpu pinning, huge
>>> pages
>>> and numa).
>>>
>>> Has anybody looked at why 1 instance is too slow and what it would
>>> take to
>>> make 1 scheduler instance work fast enough? This does not preclude the
>>> use of
>>> concurrency for finer grain tasks in the background.
>>
>> Currently we pull data on all (!) of the compute nodes out of the
>> database via a series of RPC calls, then evaluate the various filters in
>> python code.
>>
>> I suspect it'd be a lot quicker if each filter was a DB query.
>>
>> Also, ideally we'd want to query for the most "strict" criteria first,
>> to reduce the total number of comparisons. For example, if you want to
>> implement the "affinity" server group policy, you only need to test a
>> single host. If you're matching against host aggregate meta

Re: [openstack-dev] Scheduler proposal

[Alec Hothan (ahothan)]

Still the point from Chris is valid.
I guess the main reason openstack is going with multiple concurrent schedulers 
is to scale out by distributing the load between multiple instances of 
schedulers because 1 instance is too slow.
This discussion is about coordinating the many instances of schedulers in a way 
that works and this is actually a difficult problem and will get worst as the 
number of variables for instance placement increases (for example NFV is going 
to require a lot more than just cpu pinning, huge pages and numa).

Has anybody looked at why 1 instance is too slow and what it would take to make 
1 scheduler instance work fast enough? This does not preclude the use of 
concurrency for finer grain tasks in the background.




On 10/9/15, 11:05 AM, "Clint Byrum"  wrote:

>Excerpts from Chris Friesen's message of 2015-10-09 10:54:36 -0700:
>> On 10/09/2015 11:09 AM, Zane Bitter wrote:
>> 
>> > The optimal way to do this would be a weighted random selection, where the
>> > probability of any given host being selected is proportional to its 
>> > weighting.
>> > (Obviously this is limited by the accuracy of the weighting function in
>> > expressing your actual preferences - and it's at least conceivable that 
>> > this
>> > could vary with the number of schedulers running.)
>> >
>> > In fact, the choice of the name 'weighting' would normally imply that it's 
>> > done
>> > this way; hearing that the 'weighting' is actually used as a 'score' with 
>> > the
>> > highest one always winning is quite surprising.
>> 
>> If you've only got one scheduler, there's no need to get fancy, you just 
>> pick 
>> the "best" host based on your weighing function.
>> 
>> It's only when you've got parallel schedulers that things get tricky.
>> 
>
>Note that I think you mean _concurrent_ not _parallel_ schedulers.
>
>Parallel schedulers would be trying to solve the same unit of work by
>breaking it up into smaller components and doing them at the same time.
>
>Concurrent means they're just doing different things at the same time.
>
>I know this is nit-picky, but we use the wrong word _A LOT_ and the
>problem space is actually vastly different, as parallelizable problems
>have a whole set of optimizations and advantages that generic concurrent
>problems (especially those involving mutating state!) have a whole set
>of race conditions that must be managed.
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [app-catalog] PTL Candidacy

[Alec Hothan (ahothan)]

+1

Great job leading the app catalog and looking forward for all these new 
features!




On 9/11/15, 12:23 PM, "Christopher Aedo"  wrote:

>It's time for the Community App Catalog to go through an official
>election cycle, and I'm putting my name in for PTL.  I've been filling
>that role provisionally since before the App Catalog was launched at
>the Vancouver summit, and I would like to continue service as PTL
>officially.  Now that we've joined the Big Tent[1], having a committed
>leader is more important than ever :) .
>
>I believe the App Catalog has tremendous potential for helping the
>end-users of OpenStack clouds find and share things they can deploy on
>those clouds.  To that end, I've been working with folks on extending
>the types of assets that can live in the catalog and also trying to
>make finding and consuming those assets easier.
>
>Since we announced the Community App Catalog I've done everything I
>could to deliver on the "community" part.  With the help of the
>OpenStack Infra team, we moved the site to OpenStack infrastructure as
>quickly as possible.  All planning and coordination efforts have
>happened on IRC (#openstack-app-catalog), the dev and operators
>mailing list, and during the weekly IRC meetings[2].  I've also been
>working to get more people engaged and involved with the Community App
>Catalog project while attempting to raise the profile and exposure
>whenever possible.
>
>Speaking of community, I know being part of the OpenStack community at
>a broad level is one of the most important things for a PTL.  On that
>front I'm active and always available on IRC (docaedo), and do my best
>to stay on top of all the traffic on the dev mailing list.  I also
>work with Richard Raseley to organize the OpenStack meetup in Portland
>in order to reach, educate (and entertain) people who want to learn
>more about OpenStack.
>
>The next big thing we will do for the Community App Catalog is to
>build out the backend so it becomes a more engaging experience for the
>users, as well as makes it easier for other projects to contribute and
>consume the assets.  In addition to the Horizon plugin[3][4] (check it
>out with devstack, it's pretty cool!) we are thinking through the API
>side of this and will eventually contribute the code to search, fetch
>and push from the OpenStack Client.
>
>All of this is to say that I'm eager and proud to serve as the
>Community App Catalog PTL for the next six months if you'll have me!
>
>[1] https://review.openstack.org/#/c/217957/
>[2] https://wiki.openstack.org/wiki/Meetings/app-catalog
>[3] https://github.com/stackforge/apps-catalog-ui
>[4] https://github.com/openstack/app-catalog-ui
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

[Alec Hothan (ahothan)]

On 9/1/15, 11:31 AM, "gord chung" <g...@live.ca> wrote:

>
>
>On 28/08/2015 5:18 PM, Alec Hothan (ahothan) wrote:
>>
>>
>>
>>
>> On 8/28/15, 11:39 AM, "gord chung" <g...@live.ca> wrote:
>>
>>> i should start by saying i re-read my subject line and it arguably comes
>>> off aggressive -- i should probably have dropped 'explain' :)
>>>
>>> On 28/08/15 01:47 PM, Alec Hothan (ahothan) wrote:
>>>> On 8/28/15, 10:07 AM, "gord chung" <g...@live.ca> wrote:
>>>>
>>>>> On 28/08/15 12:18 PM, Roman Dobosz wrote:
>>>>>> So imagine we have new versions of the schema for the events, alarms or
>>>>>> samples in ceilometer introduced in Mitaka release while you have all
>>>>>> your ceilo services on Liberty release. To upgrade ceilometer you'll
>>>>>> have to stop all services to avoid data corruption. With
>>>>>> versionedobjects you can do this one by one without disrupting
>>>>>> telemetry jobs.
>>>>> are versions checked for every single message? has anyone considered the
>>>>> overhead to validating each message? since ceilometer is queue based, we
>>>>> could technically just publish to a new queue when schema changes... and
>>>>> the consuming services will listen to the queue it knows of.
>>>>>
>>>>> ie. our notification service changes schema so it will now publish to a
>>>>> v2 queue, the existing collector service consumes the v1 queue until
>>>>> done at which point you can upgrade it and it will listen to v2 queue.
>>>>>
>>>>> this way there is no need to validate/convert anything and you can still
>>>>> take services down one at a time. this support doesn't exist currently
>>>>> (i just randomly thought of it) but assuming there's no flaw in my idea
>>>>> (which there may be) isn't this more efficient?
>>>> If high performance is a concern for ceilometer (and it should) then maybe
>>>> there might be better options than JSON?
>>>> JSON is great for many applications but can be inappropriate for other
>>>> demanding applications.
>>>> There are other popular open source encoding options that yield much more
>>>> compact wire payload, more efficient encoding/decoding and handle
>>>> versioning to a reasonable extent.
>>> i should clarify. we let oslo.messaging serialise our dictionary how it
>>> does... i believe it's JSON. i'd be interested to switch it to something
>>> more efficient. maybe it's time we revive the msgpacks patch[1] or are
>>> there better alternatives? (hoping i didn't just unleash a storm of
>>> 'this is better' replies)
>> I'd be curious to know if there is any benchmark on the oslo serializer for 
>> msgpack and how it compares to JSON?
>> More important is to make sure we're optimizing in the right area.
>> Do we have a good understanding of where ceilometer needs to improve to 
>> scale or is it still not quite clear cut?
>
>re: serialisation, that probably isn't the biggest concern for 
>Ceilometer performance. the main items are storage -- to be addressed by 
>Gnocchi/tsdb, and polling load. i just thought i'd point out an existing 
>serialisation patch since we were on the topic :-)

Is there any data measuring the polling load on large scale deployments?
Was there a plan to reduce the polling load to an acceptable level? If yes 
could you provide any pointer if any?


>
>>
>>>> Queue based versioning might be less runtime overhead per message but at
>>>> the expense of a potentially complex queue version management (which can
>>>> become tricky if you have more than 2 versions).
>>>> I think Neutron was considering to use versioned queues as well for its
>>>> rolling upgrade (along with versioned objects) and I already pointed out
>>>> that managing the queues could be tricky.
>>>>
>>>> In general, trying to provide a versioning framework that allows to do
>>>> arbitrary changes between versions is quite difficult (and often bound to
>>>> fail).
>>>>
>>> yeah, so that's what a lot of the devs are debating about right now.
>>> performance is our key driver so if we do something we think/know will
>>> negatively impact performance, it better bring a whole lot more of
>>> something else. if queue based versioning offers comparable
>>> functionalities, i'd pers

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

[Alec Hothan (ahothan)]

On 8/28/15, 11:39 AM, gord chung g...@live.ca wrote:

i should start by saying i re-read my subject line and it arguably comes 
off aggressive -- i should probably have dropped 'explain' :)

On 28/08/15 01:47 PM, Alec Hothan (ahothan) wrote:

 On 8/28/15, 10:07 AM, gord chung g...@live.ca wrote:


 On 28/08/15 12:18 PM, Roman Dobosz wrote:
 So imagine we have new versions of the schema for the events, alarms or
 samples in ceilometer introduced in Mitaka release while you have all
 your ceilo services on Liberty release. To upgrade ceilometer you'll
 have to stop all services to avoid data corruption. With
 versionedobjects you can do this one by one without disrupting
 telemetry jobs.
 are versions checked for every single message? has anyone considered the
 overhead to validating each message? since ceilometer is queue based, we
 could technically just publish to a new queue when schema changes... and
 the consuming services will listen to the queue it knows of.

 ie. our notification service changes schema so it will now publish to a
 v2 queue, the existing collector service consumes the v1 queue until
 done at which point you can upgrade it and it will listen to v2 queue.

 this way there is no need to validate/convert anything and you can still
 take services down one at a time. this support doesn't exist currently
 (i just randomly thought of it) but assuming there's no flaw in my idea
 (which there may be) isn't this more efficient?
 If high performance is a concern for ceilometer (and it should) then maybe
 there might be better options than JSON?
 JSON is great for many applications but can be inappropriate for other
 demanding applications.
 There are other popular open source encoding options that yield much more
 compact wire payload, more efficient encoding/decoding and handle
 versioning to a reasonable extent.

i should clarify. we let oslo.messaging serialise our dictionary how it 
does... i believe it's JSON. i'd be interested to switch it to something 
more efficient. maybe it's time we revive the msgpacks patch[1] or are 
there better alternatives? (hoping i didn't just unleash a storm of 
'this is better' replies)

I'd be curious to know if there is any benchmark on the oslo serializer for 
msgpack and how it compares to JSON?
More important is to make sure we're optimizing in the right area.
Do we have a good understanding of where ceilometer needs to improve to scale 
or is it still not quite clear cut?




 Queue based versioning might be less runtime overhead per message but at
 the expense of a potentially complex queue version management (which can
 become tricky if you have more than 2 versions).
 I think Neutron was considering to use versioned queues as well for its
 rolling upgrade (along with versioned objects) and I already pointed out
 that managing the queues could be tricky.

 In general, trying to provide a versioning framework that allows to do
 arbitrary changes between versions is quite difficult (and often bound to
 fail).

yeah, so that's what a lot of the devs are debating about right now. 
performance is our key driver so if we do something we think/know will 
negatively impact performance, it better bring a whole lot more of 
something else. if queue based versioning offers comparable 
functionalities, i'd personally be more interested to explore that route 
first. is there a thread/patch/log that we could read to see what 
Neutron discovered when they looked into it?

The versioning comments are buried in this mega patch if you are brave enough 
to dig in:

https://review.openstack.org/#/c/190635

The (offline) conclusion was that this was WIP and deserved more discussion 
(need to check back with Miguel and Ihar from the Neutron team).
One option considered in that discussion was to use oslo messaging topics to 
manage flows of messages that had different versions (and still use 
versionedobjects). So if you have 3 versions in your cloud you'd end up with 3 
topics (and as many queues when it comes to Rabbit). What is complex is to 
manage the queues/topic names (how to name them), how to discover them and how 
to deal with all the corner cases (like a new node coming in with an arbitrary 
version, nodes going away at any moment, downgrade cases).


Regards,

  Alec







[1] https://review.openstack.org/#/c/151301/

-- 
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][versionedobjects][ceilometer] explain the benefits of ceilometer+versionedobjects

[Alec Hothan (ahothan)]

On 8/28/15, 10:07 AM, gord chung g...@live.ca wrote:



On 28/08/15 12:18 PM, Roman Dobosz wrote:
 So imagine we have new versions of the schema for the events, alarms or
 samples in ceilometer introduced in Mitaka release while you have all
 your ceilo services on Liberty release. To upgrade ceilometer you'll
 have to stop all services to avoid data corruption. With
 versionedobjects you can do this one by one without disrupting
 telemetry jobs.
are versions checked for every single message? has anyone considered the
overhead to validating each message? since ceilometer is queue based, we
could technically just publish to a new queue when schema changes... and
the consuming services will listen to the queue it knows of.

ie. our notification service changes schema so it will now publish to a
v2 queue, the existing collector service consumes the v1 queue until
done at which point you can upgrade it and it will listen to v2 queue.

this way there is no need to validate/convert anything and you can still
take services down one at a time. this support doesn't exist currently
(i just randomly thought of it) but assuming there's no flaw in my idea
(which there may be) isn't this more efficient?

If high performance is a concern for ceilometer (and it should) then maybe
there might be better options than JSON?
JSON is great for many applications but can be inappropriate for other
demanding applications.
There are other popular open source encoding options that yield much more
compact wire payload, more efficient encoding/decoding and handle
versioning to a reasonable extent.

Queue based versioning might be less runtime overhead per message but at
the expense of a potentially complex queue version management (which can
become tricky if you have more than 2 versions).
I think Neutron was considering to use versioned queues as well for its
rolling upgrade (along with versioned objects) and I already pointed out
that managing the queues could be tricky.

In general, trying to provide a versioning framework that allows to do
arbitrary changes between versions is quite difficult (and often bound to
fail).





 The other thing, maybe not so obvious, is to put versionedobject layer
 between application and the MongoDB driver, so that all of the schema
 changes will be automatically handled on ovo, and also serialization
 might also be done on such layer.

i don't quite understand this, is this a mongodb specific solution?
admittedly the schemaless design of mongo i can imagine causing issues
but currently we're trying to avoid wasting resources on existing
mongodb solution as we attempt to move to new api. if it's just a
generic db solution, i'd be interested to apply it to future designs.

I'd suggest to do some benchmarking to make sure the cost of
serializing/deserializing with versioned objects is not prohibitive.

  Alec



cheers,

-- 
gord


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

On 7/20/15, 5:24 AM, Bogdan Dobrelya bdobre...@mirantis.com wrote:

inline

On 14.07.2015 18:59, Alec Hothan (ahothan) wrote:
 inline...
 
 
 On 7/8/15, 8:23 AM, Bogdan Dobrelya bdobre...@mirantis.com wrote:
 
 
 I believe Oleksii is already working on it.
 
 
 
 On all above I believe it is best to keep oslo messaging simple and
 predictable, then have apps deal with any retry logic as it is really
app
 specific.

Let me not agree with the simple messaging concept. Firs of all, fault
tolerant messaging is crucial as it is a glue between all OpenStack
components (aka micro-services). Those are distributed and sometimes
statefull, therefore FT is a must at the 'app-layer', indeed.

But I strongly believe that OpenStack components should *not* repeat the
same design and code patterns to achieve FT messaging on top of the
simple Oslo libraries. That would just seem contrary to the very idea
of common messaging libraries, which Oslo project is, am I right? Or
should we instead introduce an another type of common libraries to make
messaging FT on the app lavel?

Ideally oslo messaging would provide such service however we need to
realize that not all users need the same level of fault tolerance. For
example there are many cases where it is better to see operations fail
fast instead of discovering very late about any failure (if ever, in some
cases using rabbitMQ will completely hide certain failures because of the
nature of the pub/sub service).
Even if we convince ourself that we need a unified FT messaging service,
the reality is that the current oslo messaging API description is in dire
need of fixing and the exercise of fixing it is very tricky for the
reasons discussed earlier.
Oslo messaging was great to get OpenStack rolling but we're now watching
the limits of the current codebase and it does not look like there is a
huge interest in the community to get serious in fixing it (because there
are lots of other projects in OpenStack that are more sexy for one).

This has been a long thread and maybe we should summarize what is the long
term goal and what needs to be done short term to close the gaps:
1- fix the API documentation (a tricky exercise because of the legacy
code and current active developments using oslo messaging)
2- realign drivers to comply to the new API
3- realign app code to comply to the new API

I think the plan was to create a launchpad bug and gather all the comments
we had so far in a spec and review by the community under gerrit. Is there
any better suggestion?

Thanks

  Alec

 



 Auto retries in oslo messaging can cause confusion with possible
 duplicates which could be really bad if the messages are not idempotent.
 I think trying to make oslo messaging a complex communication API is not
 realistic with the few resources available.
 It is much better to have something simple that works well (even that is
 not easy as we can see) than something complex that has lots of issues.
 
 
 
 Yes I'd like to help on that part.
 
 
 I'm glad to see more people converging on this shortcoming and the need
to
 do something.
 
 As I said above, I would keep the oslo messaging API straight and simple
 and predictable.
 The issue with that is it may make the AMQP driver non compliant as it
may
 be doing too much already but we can try to work it out.
 We should avoid having app code having to behave differently (with
if/else
 based on the driver or driver specific plugins) but maybe that will not
be
 entirely unavoidable.
 
 I'll give a short answer to all those great questions below, in the
event
 we decide to go the simple API behavior route:
 
 
 yes sender is notified about the error condition (asynchronously in the
 case of async APIs) and as quickly as possible and the app is in charge
of
 remediating to possible loss of messages (this is basically reflecting
how
 tcp or zmq unicast behaves).
 
 RabbitMQ would not comply because it would try to deliver the message
 regardless without telling the sender (until at some point it may give
up
 entirely and drop the message silently or it may try to resend forever)
or
 there exist some use cases where the message is lost (and the sender not
 notified).
 ZMQ driver is simpler because it would just reflect what the ZMQ/unicast
 library does.
 
 
 
 that would be an app bug. You don't want to send a ack before the work
is
 done and committed.
 
 
 
 that could be a oslo messaging bug if we make sure we never allow
 duplicates (that is never retry unless you are sure the recipient has
not
 already received or make sure filtering is done properly on the
receiving
 end to weed out duplicates).
 
 
 
 For CALL: use timeout and let the app remediate to it
 For CAST: leave the app remediate to it
 
 
 
 (assuming CALL) in this case let the app handle this (in general apps
will
 have to do some sort of resync with the recipients on restart if they
 care).
 
 
 I'll leave that to AMQP experts. At the oslo messaging layer I'd try to
 make it behave

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

inline...


On 7/8/15, 8:23 AM, Bogdan Dobrelya bdobre...@mirantis.com wrote:

 On 6/12/15, 3:55 PM, Clint Byrum cl...@fewbar.com wrote:
 
 
  
 
 I think you missed it is not tested in the gate as a root cause for
 some of the ambiguity. Anecdotes and bug reports are super important
for
 knowing where to invest next, but a test suite would at least
establish a
 base line and prevent the sort of thrashing and confusion that comes
from
 such a diverse community of users feeding bug reports into the system.
 
 I agree with you that zmq needs to pass whatever oslo messaging test is
 currently available however this won't remove all the
 semantical/behavioral ambiguities.
 This kind of ambiguities could be fixed by enhancing the API
documentation
 - always good to do even if a bit late - and by developing the
associated
 test cases (although they tend to be harder to write).
 Another (ugly) strategy could be to simply say that the intended
behavior
 is the one exposed by the rabbitMQ based implementation (by means of
 seniority and/or actual deployment mileage).
 
 For example, what happens if a recipient of a CALL or CAST message dies
 before the message is sent.
 Is the API supposed to return an error and if yes how quickly? RabbitMQ
 based implementation will
 likely return a success (since the message will sit in a queue in the
 broker until the consumer reconnects - which could be a long time)
while
 ZMQ based will depend on the type of pattern used. Which is the
behavior
 desired by apps and which is the behavior advertised by the oslo
 messaging API?
 
 Another example relates to flow control conditions (sender sends lots
of
 CAST, receiver very slow to consume). Should the sender
 - always receive success and all messages will be queued without limit,
 - always receive success and all messages will be queued up to a
certain
 point and new messages will be dropped silently
 - or receive an EAGAIN error (socket behavior)?
 
 In these unclear conditions, switching to a different transport driver
is
 going to be tricky because apps may have been written/patched to
assume a
 certain behavior and might no longer behave properly if the expected
 behavior changes (even if it is for the better) and may require
adjusting
 existing apps (to support a different behavior of the API).
 Note that switching to a different transport is not just about
testing
 it in devstack but also about deploying it at scale on real production
 environment and testing at scale.
 
 Alec, you bring up fantastic and importan points above.
 
 However, lets stay on track. We're not even testing to see if nova-api
 can talk to nova-conductor via the current zmq driver. It's entirely
 possible it simply does not work for any number of reasons.
 
 A devstack-gate job is the _minimum_ needed.

I believe the next steps can be summarized as the following:

1) Make existing zeromq driver tested in the gate in order to save it
from deprecation and removal.

I believe Oleksii is already working on it.



2) Think of the new driver architecture decisions more precisely, like:
- synchronous blocking REQ/REP or async DEALER/ROUTER for CALLs
- at-least-once delivery (confirms after processing) or at-most-once
(confirms before processing)
- do we want fault tolerant CALL and/or CAST, NOTIFY (AFAIK, zeromq
supports HA only for REQ/REP, ROUTER/DEALER)
- idempotent/commutative operations or ordered and non-idempotent

On all above I believe it is best to keep oslo messaging simple and
predictable, then have apps deal with any retry logic as it is really app
specific.
Auto retries in oslo messaging can cause confusion with possible
duplicates which could be really bad if the messages are not idempotent.
I think trying to make oslo messaging a complex communication API is not
realistic with the few resources available.
It is much better to have something simple that works well (even that is
not easy as we can see) than something complex that has lots of issues.


- event based notifications on changes in the numbers of clients and
servers (affects the discovery service very much)
- routing proxies and discovery service backends

Yes I'd like to help on that part.


3) Address all of the ambiguities of the API documentation in order to
keep messaging library developers and apps developers at the same
page. This is a vital step as the new driver -as well as the existing
ones- have to rely on well known and clearly described expectations and
behave appropriately.

I'm glad to see more people converging on this shortcoming and the need to
do something.

As I said above, I would keep the oslo messaging API straight and simple
and predictable.
The issue with that is it may make the AMQP driver non compliant as it may
be doing too much already but we can try to work it out.
We should avoid having app code having to behave differently (with if/else
based on the driver or driver specific plugins) but maybe that will not be
entirely unavoidable.

I'll give a short 

Re: [openstack-dev] [diskimage-builder] disk-image-create from virtualenv not working

[Alec Hothan (ahothan)]

Thanks for the tip! after a few trials I finally made it work by
commenting out the following line in /etc/sudoers:

#Defaults 
secure_path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin



  Alec


On 7/2/15, 11:25 AM, Jens Rosenboom j.rosenb...@x-ion.de wrote:

2015-07-02 19:58 GMT+02:00 Alec Hothan (ahothan) ahot...@cisco.com:

 I wonder if anybody else ran into the same issue as it looks like PyYAML
 is required on the native python in order for disk-image-crate to
succeed.
 Installing PyYAML in a virtual environment and running disk-iage-create
 from that venv won't work as one of the script fails to import yaml:


 dib-run-parts Thu Jul 2 09:27:50 PDT 2015 Running
 /tmp/image.ewtpa5DW/hooks/extra-data.d/99-squash-package-install

 ['/tmp/image.ewtpa5DW/hooks/bin', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages']

 Traceback (most recent call last):
   File
 /tmp/image.ewtpa5DW/hooks/extra-data.d/../bin/package-installs-squash,
 line 26, in module
 import yaml
 ImportError: No module named yaml


 I added a trace to print sys.path in that script and sure enough, it is
 bypassing completely my venv path, it looks like the PATH is not picking
 up the existing PATH from the calling shell.



 from the same shell that ran disk-iage-create above:
 $ pip list | grep YAML
 PyYAML (3.11)

 $ python
 Python 2.7.6 (default, Jun 22 2015, 17:58:13)
 [GCC 4.8.2] on linux2
 Type help, copyright, credits or license for more information.
 import sys
 print sys.path
 ['', '/home/localadmin/kb/lib/python2.7',
 '/home/localadmin/kb/lib/python2.7/plat-x86_64-linux-gnu',
 '/home/localadmin/kb/lib/python2.7/lib-tk',
 '/home/localadmin/kb/lib/python2.7/lib-old',
 '/home/localadmin/kb/lib/python2.7/lib-dynload', '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk',
 '/home/localadmin/kb/local/lib/python2.7/site-packages',
 '/home/localadmin/kb/lib/python2.7/site-packages']


 Is that a known issue/requirement or is there a workaround (other than
 installing pyYAML on the native python?

elements/package-installs/extra-data.d/99-squash-package-install does
a sudo -E package-installs-squash, however at least on Ubuntu the
default seems to be having env_reset defined, causing the environment
variables being cleared and thereby losing information about your
venv. As a workaround you could try to change the env_reset setting in
your /etc/sudoers.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [diskimage-builder] disk-image-create from virtualenv not working

[Alec Hothan (ahothan)]

I wonder if anybody else ran into the same issue as it looks like PyYAML
is required on the native python in order for disk-image-crate to succeed.
Installing PyYAML in a virtual environment and running disk-iage-create
from that venv won't work as one of the script fails to import yaml:


dib-run-parts Thu Jul 2 09:27:50 PDT 2015 Running
/tmp/image.ewtpa5DW/hooks/extra-data.d/99-squash-package-install

['/tmp/image.ewtpa5DW/hooks/bin', '/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages']

Traceback (most recent call last):
  File 
/tmp/image.ewtpa5DW/hooks/extra-data.d/../bin/package-installs-squash,
line 26, in module
import yaml
ImportError: No module named yaml


I added a trace to print sys.path in that script and sure enough, it is
bypassing completely my venv path, it looks like the PATH is not picking
up the existing PATH from the calling shell.



from the same shell that ran disk-iage-create above:
$ pip list | grep YAML
PyYAML (3.11)

$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type help, copyright, credits or license for more information.
 import sys
 print sys.path
['', '/home/localadmin/kb/lib/python2.7',
'/home/localadmin/kb/lib/python2.7/plat-x86_64-linux-gnu',
'/home/localadmin/kb/lib/python2.7/lib-tk',
'/home/localadmin/kb/lib/python2.7/lib-old',
'/home/localadmin/kb/lib/python2.7/lib-dynload', '/usr/lib/python2.7',
'/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk',
'/home/localadmin/kb/local/lib/python2.7/site-packages',
'/home/localadmin/kb/lib/python2.7/site-packages']


Is that a known issue/requirement or is there a workaround (other than
installing pyYAML on the native python?

Thanks

  Alec


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

Do we have a good understanding of what is expected of zmq wrt rabbitMQ?
Like in what part of the bell curve or use cases would you see it? Or
indirectly, where do we see RabbitMQ lacking today that maybe ZMQ could
handle better?
I have tried to find any information on very large scale deployment of
OpenStack and could not get a firm number (in terms of number of compute
nodes).
Does the OpenStack foundation keeps track of this information somewhere?

  Alec



On 6/19/15, 12:56 AM, Thierry Carrez thie...@openstack.org wrote:

Flavio Percoco wrote:
 There's a 95% of deployments using rabbit not because rabbit is the
 best solution for all OpenStack problems but because it was the one
 that works best now. The lack of support on other drivers caused this
 and as long this lack of support on such drivers persist, it won't
 change.

There is 95% of deployments using rabbit because RabbitMQ serves the
middle of the Bell curve of the use cases perfectly well.

For OpenStack to be ubiquitous, we need to extend beyond our comfort
zone of use cases and support technology that will let us address those.
I see zmq has an enabler for that: it will solve other classes of
problems triggered by extreme use cases.

So yes, obviously RabbitMQ dominance should (and will naturally) drive
the development and maintenance efforts. But we should at least try to
not make the lives of those who explore new trails (and reach to new use
cases) miserable.

-- 
Thierry Carrez (ttx)



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] pub/sub support for the revised zeromq driver

[Alec Hothan (ahothan)]

On 6/1/15, 5:03 PM, Davanum Srinivas 
dava...@gmail.commailto:dava...@gmail.com wrote:

fyi, the spec for zeromq driver in oslo.messaging is here:
https://review.openstack.org/#/c/187338/1/specs/liberty/zmq-patterns-usage
.rst,unified


The above spec suggests using the zmq pub/sub/xpub/xsub proxy pattern for 
implementing the oslo messaging fanout API.
This is described in the zmq guide (http://zguide.zeromq.org/, search for 
Figure 13 - Pub-Sub Network with a Proxy).

When applied to openstack, this pattern will result in a number of publishers 
connecting to a
potentially large number of proxies (the proposal calls for 1 proxy per node in 
the cloud), and each proxy connected to a relatively small number of local 
services (typically
agents).

All connections in this pattern are based on TCP (for inter-node) and could be 
based on IPC (Unix domain socket) for intra-node.
ZMQ provides the option of having either side initiate the connection, in our 
case, it is logical and simple to have the agents connect to their
local proxy (just use any well known unix domain socket name). However the 
connectivity on the other end (publisher to proxy) is not as simple without 
some form of name service/service discovery (in the present version of ZMQ 
driver, this is handled by matchmaker/redis).

GROUP MANAGEMENT
The ZMQ guide mentions rightly so that this pattern is great for decoupling 
publishers and subscribers as they can independently connect to the proxy 
without having to know each other.
So publishers would basically simply connect to *the* proxy and not have to 
deal with other publishers or any subscriber.
However when the number of proxies can be very large (thousands), the 
difficulty resides in keeping track of nodes as they can get added or removed 
during regular operations. When a node is added, we would need a way to
have each publisher connect to that new node.  Same can be said about node 
deletion.
If we go that route we will need a good solution on how to do that efficiently 
and at scale.

And this is one of my main reserve for the spec currently proposed in gerrit.

To handle group management, we could reuse the existing matchmaker code (based 
on redis) or decide to use a more streamlined solution based on a name service. 
I would like to hear about any feedback on the use of matchmaker at scale if 
there is any experience (because I do have lots of questions on how matchmaker 
is designed and how it fares regarding scale and HA).

I'll try to start another email thread regarding scale numbers (number of 
connections, current usage of fanout in the code, emulated vs. true multicast)

  Alec
















__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

Gordon,

These are all great points for RPC messages (also called CALL in oslo
messaging). There are similar ambiguous contracts for the other types of
messages (CAST and FANOUT).
I am worried about the general lack of interest from the community to fix
this as it looks like most people assume that oslo messaging is good
enough (with rabbitMQ) and hence there is no need to invest any time on an
alternative transport (not mentioning that people generally prefer to work
on newer trending areas in OpenStack than contribute on a lower-level
messaging layer).
I saw Sean Dague mention in another email that RabbitMQ is used by 95% of
OpenStack users - and therefore does it make sense to invest in ZMQ (legit
question). RabbitMQ had had a lot of issues but there has been several
commits fixing some of the issues, so it would make sense IMHO to make
another status update to reevaluate the situation.

For OpenStack to be really production grade at scale, there is a need for
a very strong messaging layer and this cannot be achieved with such a
loose API definitions (regardless of what transport is used). This will be
what distinguishes a great cloud OS platform from a so-so one.
There is also a need for defining more clearly the roadmap for oslo
messaging because it is far from over. I see a need for clarifying the
following areas:
- validation at scale and HA
- security and encryption on the control plane

  Alec



On 6/16/15, 11:25 AM, Gordon Sim g...@redhat.com wrote:

On 06/12/2015 09:41 PM, Alec Hothan (ahothan) wrote:
 One long standing issue I can see is the fact that the oslo messaging
API
 documentation is sorely lacking details on critical areas such as API
 behavior during fault conditions, load conditions and scale conditions.

I very much agree, particularly on the contract/expectations in the face
of different failure conditions. Even for those who are critical of the
pluggability of oslo.messaging, greater clarity here would be of benefit.

As I understand it, the intention is that RPC calls are invoked on a
server at-most-once, meaning that in the event of any failure, the call
will only be retried by the olso.messaging layer if it believes it can
ensure the invocation is not made twice.

If that is correct, stating so explicitly and prominently would be
worthwhile. The expectation for services using the API would then be to
decide on any retry themselves. An idempotent call could retry for a
configured number of attempts perhaps. A non-idempotent call might be
able to check the result via some other call and decide based on that
whether to retry. Giving up would then be a last resort. This would help
increase robustness of the system overall.

Again if the assumption of at-most-once is correct, and explicitly
stated, the design of the code can be reviewed to ensure it logically
meets that guarantee and of course it can also be explicitly tested for
in stress tests at the oslo.messaging level, ensuring there are no
unintended duplicate invocations. An explicit contract also allows
different approaches to be assessed and compared.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

On 6/12/15, 3:55 PM, Clint Byrum cl...@fewbar.com wrote:


 

I think you missed it is not tested in the gate as a root cause for
some of the ambiguity. Anecdotes and bug reports are super important for
knowing where to invest next, but a test suite would at least establish a
base line and prevent the sort of thrashing and confusion that comes from
such a diverse community of users feeding bug reports into the system.

I agree with you that zmq needs to pass whatever oslo messaging test is
currently available however this won't remove all the
semantical/behavioral ambiguities.
This kind of ambiguities could be fixed by enhancing the API documentation
- always good to do even if a bit late - and by developing the associated
test cases (although they tend to be harder to write).
Another (ugly) strategy could be to simply say that the intended behavior
is the one exposed by the rabbitMQ based implementation (by means of
seniority and/or actual deployment mileage).

For example, what happens if a recipient of a CALL or CAST message dies
before the message is sent.
Is the API supposed to return an error and if yes how quickly? RabbitMQ
based implementation will
likely return a success (since the message will sit in a queue in the
broker until the consumer reconnects - which could be a long time) while
ZMQ based will depend on the type of pattern used. Which is the behavior
desired by apps and which is the behavior advertised by the oslo
messaging API?

Another example relates to flow control conditions (sender sends lots of
CAST, receiver very slow to consume). Should the sender
- always receive success and all messages will be queued without limit,
- always receive success and all messages will be queued up to a certain
point and new messages will be dropped silently
- or receive an EAGAIN error (socket behavior)?

In these unclear conditions, switching to a different transport driver is
going to be tricky because apps may have been written/patched to assume a
certain behavior and might no longer behave properly if the expected
behavior changes (even if it is for the better) and may require adjusting
existing apps (to support a different behavior of the API).
Note that switching to a different transport is not just about testing
it in devstack but also about deploying it at scale on real production
environment and testing at scale.




Also, not having a test in the gate is a serious infraction now, and will
lead to zmq's removal from oslo.messaging now that we have a ratified
policy requiring this. I suggest a first step being to strive to get a
devstack-gate job that runs using zmq instead of rabbitmq. You can
trigger it in oslo.messaging's check pipeline, and make it non-voting,
but eventually it needs to get into nova, neutron, cinder, heat, etc.
etc. Without that, you'll find that the community of potential
benefactors of any effort you put into zmq will shrink dramatically when
we are forced to remove the driver from oslo.messaging (it can of course
live on out of tree).

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

On 6/1/15, 5:03 PM, Davanum Srinivas dava...@gmail.com wrote:

fyi, the spec for zeromq driver in oslo.messaging is here:
https://review.openstack.org/#/c/187338/1/specs/liberty/zmq-patterns-usage
.rst,unified

-- dims

I was about to provide some email comments on the above review off gerrit,
but figured maybe it would be good to make a quick status of the state of
this general effort for pushing out a better zmq driver for oslo essaging.
So I started to look around the oslo/zeromq wiki and saw few email threads
that drew my interest.

In this email (Nov 2014) Ilya proposes about getting rid of a central
broker for zmq:
http://lists.openstack.org/pipermail/openstack-dev/2014-November/050701.htm
l
Not clear if Ilya already had in mind to instead have a local proxy on
every node (as proposed in the above spec)


In this email (mar 2014), Yatin described the prospect of using zmq in a
completely broker-less way (so not even a proxy per node), with the use of
matchmaker rings to configure well known ports.
http://lists.openstack.org/pipermail/openstack-dev/2014-March/030411.html
Which is pretty close to what I think would be a better design (with the
variant that I'd rather see a robust and highly available name server
instead of fixed port assignments), I'd be interested to know what
happened to that proposal and why we ended up with a proxy per node
solution at this stage (I'll reply to the proxy per node design in a
separate email to complement my gerrit comments).


I could not find one document that summarizes the list of issues related
to rabbitMQ deployments, all it appears is that many people are unhappy
with it, some are willing to switch to zmq, many are hesitant and some are
decidedly skeptical. On my side I know a number of issues related to oslo
messaging over rabbitMQ.

I think it is important for the community to understand that of the many
issues generally attributed to oslo messaging over rabbitMQ, not all of
them are caused by the choice of rabbitMQ as a transport (and hence those
will likely not be fixed if we just switched from rabbitMQ to ZMQ) and
many are actually caused by the misuse of oslo messaging by the apps
(Neutron, Nova...) and can only be fixed by modification of the app code.

I think personally that there is a strong case for a properly designed ZMQ
driver but we first need to make the expectations very clear.

One long standing issue I can see is the fact that the oslo messaging API
documentation is sorely lacking details on critical areas such as API
behavior during fault conditions, load conditions and scale conditions.
As a result, app developers are using the APIs sometimes indiscriminately
and that will have an impact on the overall quality of openstack in
deployment conditions.
I understand that a lot of the existing code was written in a hurry and
good enough to work properly on small setups, but some code will break
really badly under load or when things start to go south in the cloud.
That is unless the community realizes that perhaps there is something that
needs to be done.

We're only starting to see today things breaking under load because we
have more lab tests at scale, more deployments at scale and we only start
to see real system level testing at scale with HA testing (the kind of
test where you inject load and cause failures of all sorts). Today we know
that openstack behaves terribly in these conditions, even in so-called HA
deployments!

As a first step, would it be useful to have one single official document
that characterizes all the issues we're trying to fix and perhaps used
that document as a basis for showing which of all these issues will be
fixed by the use of the zmq driver? I think that could help us focus
better on the type of requirements we need from this new ZMQ driver.


Thanks,

  Alec




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

).



I'll cite the zmq-guide about broker/brokerless (4.14. Brokerless Reliability 
p.221):

It might seem ironic to focus so much on broker-based reliability, when we 
often explain ØMQ as brokerless messaging. However, in messaging, as in real 
life, the middleman is both a burden and a benefit. In practice, most messaging 
architectures benefit from a mix of distributed and brokered messaging. 

Brokers and middlemen are beneficial in many situations no question about it. 
In this particular situation there is actually already a broker of sort which 
is the redis server ;-) The redis server acts like a name server and allows 
dynamic discovery of services (topics and the associated addresses).
Brokers are interesting for 2 reasons:

  *   decouple participants of a communication infra (e.g. you do not want to 
hardcode anything about peers, their count and their addresses), this can be 
done by a broker a la 0MQ example or by a name server
  *   do something special about messages that are being brokered (e.g. 
persistence, replication, multicast, load balancing etc...), things that you 
can't do with simple peer to peer connections and things where RabbitMQ 
(presumably) excels

Given that the proxy server does not seem to do anything special with the 
messages (other than forwarding/unicasting) and given that Redis could provide 
a full end to end addressing, it seems that the need for a proxy is greatly 
diminished.
A list of drawbacks in having a proxy server in every node:

  *   potential deployment complications as noted above,
  *   total connection count is higher compared to proxy-less design - this 
could be a problem if we ever get to the point of encrypting every connection 
(btw 0MQ supports encryption since version 3 although I have not personally 
tried it)
  *   1 more hop for every message in both directions
  *   not clear if the proxy would propagate disconnection events from one side 
to the other?
  *   need to tend to the buffering and flow control in the proxy (one policy 
may not fit all needs and will it still be state-less)

Lastly, note that the Redis server itself could be clustered for HA (a feature 
added recently) and this might be something we may have to look at as well 
because it is another point of failure (it would be awkward to put the redis 
server on 1 controller node where HA calls for 3 controller nodes for example).

I'm still relatively new to oslo messaging and still have a lot of questions 
regarding a deployment based on 0MQ. I think it is important that we assess 
properly the forces in favor of this protocol and make sure it does provide a 
better option than rabbitMQ at production scale using measurable evidence.

Thanks

  Alec







Thanks,
Oleksii


1 - http://zeromq.org/area:faq#toc7


5/26/15 18:57, Davanum Srinivas пишет:

Alec,

Here are the slides:
http://www.slideshare.net/davanum/oslomessaging-new-0mq-driver-proposal

All the 0mq patches to date should be either already merged in trunk
or waiting for review on trunk.

Oleksii, Li Ma,
Can you please address the other questions?

thanks,
Dims

On Tue, May 26, 2015 at 11:43 AM, Alec Hothan (ahothan)
ahot...@cisco.commailto:ahot...@cisco.com wrote:


Looking at what is the next step following the design summit meeting on
0MQ as the etherpad does not provide too much information.
Few questions:
- would it be possible to have the slides presented (showing the proposed
changes in the 0MQ driver design) to be available somewhere?
- is there a particular branch in the oslo messaging repo that contains
0MQ related patches - I'm more particularly interested by James Page's
patch to pool the 0MQ connections but there might be other
- question for Li Ma, are you deploying with the straight upstream 0MQ
driver or with some additional patches?

The per node proxy process (which is itself some form of broker) needs to
be removed completely if the new solution is to be made really
broker-less. This will also eliminate the only single point of failure in
the path and reduce the number of 0MQ sockets (and hops per message) by
half.

I think it was proposed that we go on with the first draft of the new
driver (which still keeps the proxy server but reduces the number of
sockets) before eventually tackling the removal of the proxy server?



Thanks

  Alec



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribemailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [oslo.messaging][zeromq] Next step

[Alec Hothan (ahothan)]

Looking at what is the next step following the design summit meeting on
0MQ as the etherpad does not provide too much information.
Few questions:
- would it be possible to have the slides presented (showing the proposed
changes in the 0MQ driver design) to be available somewhere?
- is there a particular branch in the oslo messaging repo that contains
0MQ related patches - I'm more particularly interested by James Page's
patch to pool the 0MQ connections but there might be other
- question for Li Ma, are you deploying with the straight upstream 0MQ
driver or with some additional patches?

The per node proxy process (which is itself some form of broker) needs to
be removed completely if the new solution is to be made really
broker-less. This will also eliminate the only single point of failure in
the path and reduce the number of 0MQ sockets (and hops per message) by
half.

I think it was proposed that we go on with the first draft of the new
driver (which still keeps the proxy server but reduces the number of
sockets) before eventually tackling the removal of the proxy server?



Thanks

  Alec



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [devstack]Specific Juno version

[Alec Hothan (ahothan)]

You can specify specific tags for each component in your local.conf file, e.g.:

NOVA_BRANCH=2014.2
CINDER_BRANCH=2014.2
GLANCE_BRANCH=2014.2
HORIZON_BRANCH=2014.2
KEYSTONE_BRANCH=2014.2
KEYSTONECLIENT_BRANCH=2014.2
NOVACLIENT_BRANCH=2014.2
SWIFT_BRANCH=2014.2
HEAT_BRANCH=2014.2

HTH

  Alec


From: Eduard Matei 
eduard.ma...@cloudfounders.commailto:eduard.ma...@cloudfounders.com
Reply-To: OpenStack Development Mailing List (not for usage questions) 
openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org
Date: Tuesday, March 3, 2015 at 7:42 AM
To: OpenStack Development Mailing List (not for usage questions) 
OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org
Subject: [openstack-dev] [devstack]Specific Juno version

Hi,

Is there a way to specify the Juno version to be installed using devstack.
For now we can only specify stable/juno
git clone -b stable/juno https://github.com/openstack-dev/devstack.git 
/opt/stack/devstack
but this installs 2014.2.3 which appears to be still under development (it 
gives some errors).
How can we specify 2014.2.2 for components (e.g. cinder)?

Thanks,
Eduard

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev