date:20120229

OK, first version of Debianized nova-docs, for review:

[PATCH] Docs: Debianizing The Manuals, v01
https://bugs.launchpad.net/openstack-manuals/+bug/943792

Where are the sources of the Starter Guide ?

myuser@nova-server1:~/openstack-manuals/doc/src/docbkx/openstack-compute-admin$
ls ..
api-quick-startopenstack-identity-service-starter
common openstack-image-service-admin
docbkx-example openstack-install
openstack-api-programming  openstack-network-connectivity-admin
openstack-api-site openstack-object-storage-admin
openstack-compute-admin

===
PATCH:
+++ computeinstall.xml  2012-02-29 23:25:44.0 -0500
@@ -127,6 +127,46 @@
 clicking the links next to the manual title.
 
 
+
+
+Installing OpenStack Compute on Debian
+Starting with Debian 7.0 "Wheezy", the OpenStack packages are
+provided as part of the distribution.
+
+  For the management node install the following packages:
(via apt-get install)
+
+nova-api
+nova-scheduler
+glance
+keystone
+mysql-server
+rabbitmq
+memcached
+openstack-dashboard
+
+
+
+  For the compute node(s) install the following packages:
+
+nova-compute
+nova-network
+nova-api
+
+
+  
+
+  Because this manual takes active advantage of the "sudo" command,
+  it would be easier for you to add to it your Debian
system, by doing:
+  
+# usermod -a -G sudo "myuser"
+  
+  then re-login.
+  Otherwise you will have to replace every "sudo" call by
executing from
+  root account.
+  
+
+
+
 
 Installing OpenStack Compute on Red Hat Enterprise
Linux 6 
 This section documents a multi-node installation using
RHEL 6. RPM repos for the Bexar

-- 
-Alexey Eromenko "Technologov"

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] If I open a bug in launchpad + attach a patch, does it get reviewed ?

2012-02-29 Thread Lorin Hochstein

No, it only shows up in the Gerrit queue   if you submit 
the patch using git.

Take care,

Lorin
--
Lorin Hochstein
Lead Architect - Cloud Services
Nimbis Services, Inc.
www.nimbisservices.com





On Feb 29, 2012, at 9:13 PM, Alexey Eromenko wrote:

> If I open a bug in launchpad + attach a patch, does it get into a
> review queue automatically ?
> (so far I patch docs)
> 
> -- 
> -Alexey Eromenko "Technologov"
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp



smime.p7s
Description: S/MIME cryptographic signature
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Debianizing Documentation

Hi All !

...besides the patches I already sent,
I'm currently working on Debianizing the official documentation,
openstack-manuals.

I will provide a non-intrusive patch, by adding "sudo" support to Debian:
# usermod -a -G sudo 

+ explain the basic packages, that need be installed, plus add mention
of Debian across the docs.
I have started with "staring guide" and "nova" manuals.
This approach will allow to keep much of the guide pre-pended by "sudo
this; sudo that", at the same time supporting Debian...

For the curious: Debian project *has* own HOW-TO here:
http://wiki.debian.org/OpenStackHowto

... but I believe it is important to embrace Debian a bit more boldly
within the OpenStack community, at the official docs level.
-- 
-Alexey Eromenko "Technologov"

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] eventlet weirdness

On Wed, Feb 29, 2012, Yun Mao  wrote:
> Thanks for the explanation. Let me see if I understand this.
> 
> 1. Eventlet will never have this problem if there is only 1 OS thread
> -- let's call it main thread.

In fact, that's exactly what Python calls it :)

> 2. In Nova, there is only 1 OS thread unless you use xenapi and/or the
> virt/firewall driver.
> 3. The python logging module uses locks. Because of the monkey patch,
> those locks are actually eventlet or "green" locks and may trigger a
> green thread context switch.
> 
> Based on 1-3, does it make sense to say that in the other OS threads
> (i.e. not main thread), if logging (plus other pure python library
> code involving locking) is never used, and we do not run a eventlet
> hub at all, we should never see this problem?

That should be correct. I'd have to double check all of the monkey
patching that eventlet does to make sure there aren't other cases where
you may inadvertently use eventlet primitives across real threads.

JE


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] If I open a bug in launchpad + attach a patch, does it get reviewed ?

If I open a bug in launchpad + attach a patch, does it get into a
review queue automatically ?
(so far I patch docs)

-- 
-Alexey Eromenko "Technologov"

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] eventlet weirdness

Thanks for the explanation. Let me see if I understand this.

1. Eventlet will never have this problem if there is only 1 OS thread
-- let's call it main thread.
2. In Nova, there is only 1 OS thread unless you use xenapi and/or the
virt/firewall driver.
3. The python logging module uses locks. Because of the monkey patch,
those locks are actually eventlet or "green" locks and may trigger a
green thread context switch.

Based on 1-3, does it make sense to say that in the other OS threads
(i.e. not main thread), if logging (plus other pure python library
code involving locking) is never used, and we do not run a eventlet
hub at all, we should never see this problem?

Thanks,

Yun

On Wed, Feb 29, 2012 at 5:24 PM, Johannes Erdfelt  wrote:
> On Wed, Feb 29, 2012, Yun Mao  wrote:
>> we sometimes notice this error message which prevent us from starting
>> nova services occasionally. We are using a somewhat modified diablo
>> stable release on Ubuntu 11.10. It may very well be the problem from
>> our patches but I'm wondering if you guys have any insight. In what
>> condition does this error occur? There is a similar bug in here:
>> https://bugs.launchpad.net/nova/+bug/831599
>>
>> but that doesn't offer much insight to me. Helps are very appreciated. 
>> Thanks,
>
> greenlet threads (used by eventlet) can't be scheduled across real
> threads. This usually isn't done explicitly, but can happen as a side
> effect if code uses locks. logging is one instance that I've run into.
>
> This generally hasn't been a problem with nova since it uses the
> eventlet monkey patching that makes it hard to generate real threads.
>
> There are two places (at least in trunk) where you need to be careful,
> both nova/virt/xenapi_conn.py and libvirt/firewall.py use tpool which
> does create a real thread in the background.
>
> If you use logging (and it's not the only source of this problem) then
> you can run into this eventlet message.
>
> JE
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Docs: Would like to introduce indexing. How ?

On Tue, Feb 28, 2012 at 4:46 AM, Anne Gentle  wrote:
> It's not a "dislike" - rather it fulfills a need for reuse of content
> so that sections are not rigidly numbered for certain types of
> deliverables, such as the admin guides.

reuse is OK.
I speak about dynamically-generated indexes, by the PDF / HTML compilers.
So only final docs will have them, not XML src code.

-- 
-Alexey Eromenko "Technologov"

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Docs: Lack of OpenStack storage concepts explanation - Please help to define

Docs: Lack of OpenStack storage concepts explanation
https://bugs.launchpad.net/openstack-manuals/+bug/938365

Please help to better define the concepts.
... I still don't really know what's the diff between nova-volume and
nova-object.
as well as other concepts.

We should propose several definitions of the following:
-nova-volume
-nova-object
-swift
-glance

Then choose whatever is best.
It must be understandable by people familiar only with NFS / block storage.
-- 
-Alexey Eromenko "Technologov"

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Swift client tool

2012-02-29 Thread Pratt, Gavin

CloudBerry works as well
http://www.cloudberrylab.com/

From: openstack-bounces+gavin.pratt=hp@lists.launchpad.net 
[mailto:openstack-bounces+gavin.pratt=hp@lists.launchpad.net] On Behalf Of 
Kuo Hugo
Sent: Monday, December 19, 2011 11:15 PM
To: Tim Bell
Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Swift client tool

As I know . both cyberduck and gladinet provide this feature .
But gladinet require pro. version to enable sync and backup solution with swift

2011/12/20 Tim Bell mailto:tim.b...@cern.ch>>

A Dropbox like 'sync' function would be very interesting..  does anyone know 
one which is compatible with OpenStack Swift ?

Tim

From: 
openstack-bounces+tim.bell=cern...@lists.launchpad.net

[mailto:openstack-bounces+tim.bell=cern...@lists.launchpad.net]
 On Behalf Of Kuo Hugo
Sent: 19 December 2011 17:38
To: Prakashan Korambath

Cc: openstack@lists.launchpad.net
Subject: Re: [Openstack] Swift client tool

Several options

1. Cyberduck (for Mac & Win only) , swift will present like a FTP server user 
experience for you
2. Gladinet desktop (free version) , under gladinet , you might get feel swift 
more like a NAS device ... but only for static object files , not that easy to 
setup a gladinet compatible swift environment . It requires SSL and validate 
SSL certification .
Only for Win OS
3. Under Linux , you can leverage swift client , the easiest way is #apt-get 
install swift
4. Write your own client by call swift client module
5. Write your own client through swift API endpoint
6. Using OpenStack Dashboard , it includes Swift feature. but it requires 
keystone integration
7. develop your own Web server for access Swift

We can confirm all approaches above .  but might need to dig out some more 
tricky skill from google .

If your swift only for personal usage , you can easily install cyberduck to 
access swift. In my using , I just need to setup auth server endpoint manually 
in cyberduck's configuration file to point the correct auth server endpoint  
which depends on your auth server .
more information plz goole it .
Feel free to drop your question over here . I'll have an answer for you as I 
can.

+Hugo Kuo+
tonyt...@gmail.com
hugo@cloudena.com
+886-935-004-793

www.cloudena.com

--
+Hugo Kuo+
tonyt...@gmail.com
hugo@cloudena.com
+886-935-004-793

www.cloudena.com
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Memory leaks from greenthreads

2012-02-29 Thread Joshua Harlow

I agree its not trivial, but it might be worth it?

Or at least shift the model toward something semi-equivalent (multiprocessing??)

If the python community says though that better inherent support for eventlet 
will arrive (ie something in the core python) someday then I say leave it. 
Otherwise I guess if the problems increase to a intolerable amount then 
something might have to happen (hopefully earlier rather than later).

On 2/29/12 2:32 PM, "Johannes Erdfelt"  wrote:

On Wed, Feb 29, 2012, Joshua Harlow  wrote:
> Just a thought I was having, that others might want to chime in on.
>
> Has there been any thinking around only using eventlet/greenlet for
> webserver endpoints and using something like multiprocessing for
> everything else?
>
> I know its a fundamental change, but it would force people to think
> about how to break up there code into something that would work with
> a message passing architecture (this is already happening with
> nova + rabbitmq). Nova is a good example, but my thought was to go even
> further and have anything that needs to run for a long time (ie a
> equivalent of a nova manager) that is shared inside a application also
> be a separate "process" with a queue for message passing. Then maybe
> eventlet/greenlet isn't needed at all? This would force good
> interfaces, and we wouldn't have to worry about missing a monkey patch.
> Maybe the python people plan for multiprocess to replace
> eventlet/greenlet in the end anyway???

I personally would be for an effort to remove eventlet. It does a pretty
good job, but monkey patching is error prone. For instance, my patch for
this eventlet memory leak also had to workaround lockfile patching the
threading.Thread object.

The biggest problem is how the exception context gets cleared if
eventlet threads get switched. It's very hard to determine what
exception handlers need a sprinkling of save_and_reraise_exception() to
workaround that issue. That isn't the only reason for that function,
but it's difficult to know when it's necessary.

That said, the work won't be trivial, especially with things like
connection pooling for XenAPI connections.

JE

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] eventlet weirdness

2012-02-29 Thread Michael Pittaro

On Wed, Feb 29, 2012 at 1:02 PM, Yun Mao  wrote:
> Hi,
>
> we sometimes notice this error message which prevent us from starting
> nova services occasionally. We are using a somewhat modified diablo
> stable release on Ubuntu 11.10. It may very well be the problem from
> our patches but I'm wondering if you guys have any insight. In what
> condition does this error occur? There is a similar bug in here:
> https://bugs.launchpad.net/nova/+bug/831599
>
> but that doesn't offer much insight to me. Helps are very appreciated. Thanks,
>
> Yun

One tip - make sure you capture stdout/stderr, as well as logs.

Although I haven't see this particular error, I have seen at least
one case where libvirt related errors weren't in the log, but made
it to the console.

mike

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Memory leaks from greenthreads

On Wed, Feb 29, 2012, Joshua Harlow  wrote:
> Just a thought I was having, that others might want to chime in on.
> 
> Has there been any thinking around only using eventlet/greenlet for
> webserver endpoints and using something like multiprocessing for
> everything else?
> 
> I know its a fundamental change, but it would force people to think
> about how to break up there code into something that would work with
> a message passing architecture (this is already happening with
> nova + rabbitmq). Nova is a good example, but my thought was to go even
> further and have anything that needs to run for a long time (ie a
> equivalent of a nova manager) that is shared inside a application also
> be a separate "process" with a queue for message passing. Then maybe
> eventlet/greenlet isn't needed at all? This would force good
> interfaces, and we wouldn't have to worry about missing a monkey patch.
> Maybe the python people plan for multiprocess to replace
> eventlet/greenlet in the end anyway???

I personally would be for an effort to remove eventlet. It does a pretty
good job, but monkey patching is error prone. For instance, my patch for
this eventlet memory leak also had to workaround lockfile patching the
threading.Thread object.

The biggest problem is how the exception context gets cleared if
eventlet threads get switched. It's very hard to determine what
exception handlers need a sprinkling of save_and_reraise_exception() to
workaround that issue. That isn't the only reason for that function,
but it's difficult to know when it's necessary.

That said, the work won't be trivial, especially with things like
connection pooling for XenAPI connections.

JE

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] eventlet weirdness

On Wed, Feb 29, 2012, Yun Mao  wrote:
> we sometimes notice this error message which prevent us from starting
> nova services occasionally. We are using a somewhat modified diablo
> stable release on Ubuntu 11.10. It may very well be the problem from
> our patches but I'm wondering if you guys have any insight. In what
> condition does this error occur? There is a similar bug in here:
> https://bugs.launchpad.net/nova/+bug/831599
> 
> but that doesn't offer much insight to me. Helps are very appreciated. Thanks,

greenlet threads (used by eventlet) can't be scheduled across real
threads. This usually isn't done explicitly, but can happen as a side
effect if code uses locks. logging is one instance that I've run into.

This generally hasn't been a problem with nova since it uses the
eventlet monkey patching that makes it hard to generate real threads.

There are two places (at least in trunk) where you need to be careful,
both nova/virt/xenapi_conn.py and libvirt/firewall.py use tpool which
does create a real thread in the background.

If you use logging (and it's not the only source of this problem) then
you can run into this eventlet message.

JE

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] please compress the public nova.git repository (~100MB -> 30MB)

2012-02-29 Thread James E. Blair

Jim Meyering  writes:

> Future cloners (and those who push, too) will be
> grateful if someone with access to the server would
> do the same thing to the public nova.git repository.
> Compressing the repo improves server performance, too.

Hi,

That's a good idea, thanks for the suggestion!  I want to do a little
more research and testing, but it looks like a periodic process like
this on the gerrit server would be a good idea.  I expect we'll do
something like what you suggest soon.

I filed https://bugs.launchpad.net/openstack-ci/+bug/943623 for this.

-Jim

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] please compress the public nova.git repository (~100MB -> 30MB)

2012-02-29 Thread Jim Meyering

Joe Gordon wrote:
> If you just want to install from source or just look at the current code you 
> can
> do a:
>
> git clone git://github.com/openstack/nova.git --depth 1

Thanks for the tip, but I do know about git's --depth option.

The point of my message was to encourage someone with server access
to make it so every full clone is 70MB smaller (also makes the server
more efficient for fetch and push).
Less time to download and no need for every developer to
compress his/her own copy -- assuming they think to do that.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] eventlet weirdness

2012-02-29 Thread Mark Gius

I have been encountering these quite a bit myself recently in another
project.

For me the errors were a result of tpool.execute() in a non-cooperative
thread context.  My guess as to the root cause is that some of eventlet's
cooperative waiting code is not safe to use when not running in an eventlet
coroutine context.  My solution (which may not work for you) involved
switching based on whether I'm in a greenthread or not, and either call
tpool.execute() or the underlying function directly.  Fortunately for me I
can know at compile time what context I will be in. I think there is a way
to query eventlet to see if you are currently in a greenthread or not, but
I haven't finished diving into that documentation yet.

Good luck,
Mark

On Wed, Feb 29, 2012 at 1:02 PM, Yun Mao  wrote:

> Hi,
>
> we sometimes notice this error message which prevent us from starting
> nova services occasionally. We are using a somewhat modified diablo
> stable release on Ubuntu 11.10. It may very well be the problem from
> our patches but I'm wondering if you guys have any insight. In what
> condition does this error occur? There is a similar bug in here:
> https://bugs.launchpad.net/nova/+bug/831599
>
> but that doesn't offer much insight to me. Helps are very appreciated.
> Thanks,
>
> Yun
>
> 2012-02-23 16:54:52,788 DEBUG nova.utils
> [43f98259-6ba8-4e5d-bc0e-9eab978194e5 None None] backend  'nova.db.sqlalchemy.api' from
> '/opt/stack/nova/nova/db/sqlalchemy/api.pyc'> from (pid=6385)
> __get_backend /opt/stack/nova/nova/utils.py:449
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line
> 336, in fire_timers
>timer()
>  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line
> 56, in __call__
>cb(*args, **kw)
>  File "/usr/lib/python2.7/dist-packages/eventlet/semaphore.py", line
> 95, in _do_acquire
>waiter.switch()
> error: cannot switch to a different thread
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Memory leaks from greenthreads

2012-02-29 Thread Joshua Harlow

Cool.

Just a thought I was having, that others might want to chime in on.

Has there been any thinking around only using eventlet/greenlet for webserver
endpoints and using something like multiprocessing for everything else?

I know its a fundamental change, but it would force people to think about how
to break up there code into something that would work with a message passing
architecture (this is already happening with nova + rabbitmq). Nova is a good
example, but my thought was to go even further and have anything that needs to
run for a long time (ie a equivalent of a nova manager) that is shared inside a
application also be a separate "process" with a queue for message passing. Then
maybe eventlet/greenlet isn't needed at all? This would force good interfaces,
and we wouldn't have to worry about missing a monkey patch. Maybe the python
people plan for multiprocess to replace eventlet/greenlet in the end anyway???

Thoughts?

On 2/29/12 12:48 PM, "Vishvananda Ishaya" wrote:

Hello Everyone,

We have had a memory leak due to an interaction with eventlet for a while that
Johannes has just made a fix for.

bug:
https://bugs.launchpad.net/nova/+bug/903199

fix:
https://bitbucket.org/which_linden/eventlet/pull-request/10/monkey-patch-threadingcurrent_thread-as

Unfortuantely, I don' t think we have a decent workaround for nova while that
patch is upstreamed. I wanted to make sure that all of the distros are aware
of it in case they want to carry an eventlet patch to prevent the slow memory
leak.

Vish

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp

Re: [Openstack] how to run selected tests

2012-02-29 Thread Andrew Clay Shafer

The way run_tests.sh works right now, you can run all tests, all tests in a
file, a module or an individual test depending on what args you run with.

The default with no args will run all the tests.

You can run one file passing in the name of the file minus .py (if it is a
sub-directory, replace slashes with .)

./run_tests.sh test_compute

or

./run_tests.sh scheduler.test_scheduler

A module can be run passing in the file and the module:

./run_tests.sh scheduler.test_scheduler:SchedulerManagerTestCase

A single test by adding the name of the test:

./run_tests.sh
scheduler.test_scheduler:SchedulerManagerTestCase.test_existing_method

Hope that helps.




On Wed, Feb 29, 2012 at 3:42 PM, Yun Mao  wrote:

> Greetings,
>
> What's the most convenient way to run a subset of the existing tests?
> By default run_tests.sh tests everything. For example, I'd like to run
> everything in test_scheduler plus test_notify.py, what's the best way
> to do that? Thanks,
>
> Yun
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] how to run selected tests

2012-02-29 Thread Adrian Smith

nova/testing/runner.py has a few suggestions in the header comments.

e.g.

To run a single test module:
python nova/testing/runner.py test_compute


On 29 February 2012 20:42, Yun Mao  wrote:
> Greetings,
>
> What's the most convenient way to run a subset of the existing tests?
> By default run_tests.sh tests everything. For example, I'd like to run
> everything in test_scheduler plus test_notify.py, what's the best way
> to do that? Thanks,
>
> Yun
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] Memory leaks from greenthreads

On Wed, Feb 29, 2012, Vishvananda Ishaya  wrote:
> We have had a memory leak due to an interaction with eventlet for a
> while that Johannes has just made a fix for.
> 
> bug:
> https://bugs.launchpad.net/nova/+bug/903199
> 
> fix:
> https://bitbucket.org/which_linden/eventlet/pull-request/10/monkey-patch-threadingcurrent_thread-as
> 
> Unfortuantely, I don' t think we have a decent workaround for nova
> while that patch is upstreamed.  I wanted to make sure that all of
> the distros are aware of it in case they want to carry an eventlet
> patch to prevent the slow memory leak.

There is one other possible workaround, but I didn't feel like it was
safe since we would likely need to audit all of the third party libraries
to ensure they don't cause problems.

The memory leak only happens when monkey patching the
thread/threading/Queue modules. I looked at the nova sources and did
some tests and it doesn't appear nova needs those modules patches.

However, third party modules might need it. Also, I only tested on
xenapi. libvirt and/or vmwareapi might have problems. Or possibly other
drivers (firewall, volume, etc) in nova that I didn't use in my tests.

If you're having problems with the memory leak in eventlet and applying
the patch isn't an option, then monkey patching everything but thread
might something worth trying.

eventlet.monkey_patch(os=True, socket=True, time=True)

JE

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] please compress the public nova.git repository (~100MB -> 30MB)

2012-02-29 Thread Joe Gordon

If you just want to install from source or just look at the current code
you can do a:

git clone git://github.com/openstack/nova.git --depth 1

best,
Joe

On Tue, Feb 28, 2012 at 8:06 AM, Jim Meyering  wrote:

> Hello,
>
> Looking at nova.git for the first time, I cloned it.
> Surprised that it was so much larger than the others, and too large for
> the size of the source and the amount of history, I tried to compress it.
> That reduced the repository size from nearly 100MiB to just 30:
>
>  $ git clone git://github.com/openstack/nova.git
>  $ git-repo-compress .git
>  97M .git
>  Counting objects: 101858, done.
>  Delta compression using up to 12 threads.
>  Compressing objects: 100% (100594/100594), done.
>  Writing objects: 100% (101858/101858), done.
>  Total 101858 (delta 80708), reused 18388 (delta 0)
>  150.91user 81.16system 0:35.05elapsed 662%CPU (0avgtext+0avgdata
> 324536maxresident)k
>  29056inputs+59312outputs (381major+252932minor)pagefaults 0swaps
>  started Tue 2012-02-28 16:40:42 +0100
>  Tue 2012-02-28 16:41:17 +0100
>  30M .git
>
> The command I used is just a wrapper around git repack:
>
>  git-repo-compress () {
>  local d=$1
>  du -sh "$d"
>  start=$(date)
>  env time git --git-dir="$d" repack -afd --window=250 --depth=250
>  echo started $start
>  date
>  du -sh "$d"
>  }
>
> Future cloners (and those who push, too) will be
> grateful if someone with access to the server would
> do the same thing to the public nova.git repository.
> Compressing the repo improves server performance, too.
>
> These other repositories compressed well, too:
>  size of .git repo (MiB)
>   current  compressed
>  swift  9.8M2.3M
>  keystone11M9.9M
>  horizon4.1M3.2M
>  glance 5.2M1.9M
>  quantum3.0M1.4M
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] how to run selected tests

Cool. Thanks Brad and Vish for the quick reply!

Yun

On Wed, Feb 29, 2012 at 3:57 PM, Vishvananda Ishaya
 wrote:
> ./run_tests.sh -N scheduler test_notify
> (replace -N with -V if you are using a virtual environment)
> You can also get very specific using : for class separation and . for path 
> and method separation
>
> for example:
> ./run_tests.sh -N 
> nova.tests.api.openstack.compute.contrib.test_floating_ips:FloatingIpSerializerTest.test_index_serializer
>
> also consider using -n for subsequent runs of run_tests.sh, it stops 
> run_tests.sh from recreating the database each time
>
> Vish
>
> On Feb 29, 2012, at 12:42 PM, Yun Mao wrote:
>
>> Greetings,
>>
>> What's the most convenient way to run a subset of the existing tests?
>> By default run_tests.sh tests everything. For example, I'd like to run
>> everything in test_scheduler plus test_notify.py, what's the best way
>> to do that? Thanks,
>>
>> Yun
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] eventlet weirdness

Hi,

we sometimes notice this error message which prevent us from starting
nova services occasionally. We are using a somewhat modified diablo
stable release on Ubuntu 11.10. It may very well be the problem from
our patches but I'm wondering if you guys have any insight. In what
condition does this error occur? There is a similar bug in here:
https://bugs.launchpad.net/nova/+bug/831599

but that doesn't offer much insight to me. Helps are very appreciated. Thanks,

Yun

2012-02-23 16:54:52,788 DEBUG nova.utils
[43f98259-6ba8-4e5d-bc0e-9eab978194e5 None None] backend  from (pid=6385)
__get_backend /opt/stack/nova/nova/utils.py:449
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line
336, in fire_timers
timer()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line
56, in __call__
cb(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/eventlet/semaphore.py", line
95, in _do_acquire
waiter.switch()
error: cannot switch to a different thread

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] how to run selected tests

2012-02-29 Thread Vishvananda Ishaya

./run_tests.sh -N scheduler test_notify
(replace -N with -V if you are using a virtual environment)
You can also get very specific using : for class separation and . for path and 
method separation

for example:
./run_tests.sh -N 
nova.tests.api.openstack.compute.contrib.test_floating_ips:FloatingIpSerializerTest.test_index_serializer

also consider using -n for subsequent runs of run_tests.sh, it stops 
run_tests.sh from recreating the database each time

Vish

On Feb 29, 2012, at 12:42 PM, Yun Mao wrote:

> Greetings,
> 
> What's the most convenient way to run a subset of the existing tests?
> By default run_tests.sh tests everything. For example, I'd like to run
> everything in test_scheduler plus test_notify.py, what's the best way
> to do that? Thanks,
> 
> Yun
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] how to run selected tests

2012-02-29 Thread Jay Pipes


On 02/29/2012 03:42 PM, Yun Mao wrote:

Greetings,

What's the most convenient way to run a subset of the existing tests?
By default run_tests.sh tests everything. For example, I'd like to run
everything in test_scheduler plus test_notify.py, what's the best way
to do that? Thanks,


jpipes@uberbox:~/repos/nova$ ./run_tests.sh 
nova.tests.scheduler.test_scheduler nova.tests.notifier

No virtual environment found...create one? (Y/n) n
cleaning *.pyc files
SchedulerDriverModuleTestCase
test_cast_to_compute_host_no_update_db  OK  0.00
test_cast_to_compute_host_update_db_with_instance_idOK  0.00
test_cast_to_compute_host_update_db_without_instance_id OK  0.00
test_cast_to_host_compute_topic OK  0.00
test_cast_to_host_network_topic OK  0.00
test_cast_to_host_unknown_topic OK  0.00
test_cast_to_host_volume_topic  OK  0.00
test_cast_to_network_host   OK  0.00
test_cast_to_volume_host_no_update_db   OK  0.00
test_cast_to_volume_host_update_db_with_volume_id   OK  0.00
test_cast_to_volume_host_update_db_without_volume_idOK  0.00
test_encode_instanceOK  0.00
SchedulerManagerTestCase
test_1_correct_init OK  0.00
test_existing_methodOK  0.00
test_get_host_list  OK  0.00
test_get_service_capabilities   OK  0.00
test_missing_method_fallbackOK  0.00
test_run_instance_exception_puts_instance_in_error_stateOK  0.00
test_show_host_resourcesOK  0.00
test_update_service_capabilitiesOK  0.00
SchedulerTestCase
test_block_migration_dest_check_service_lack_disk   OK  0.00
test_create_instance_db_entry   OK  0.00
test_get_host_list  OK  0.00
test_get_service_capabilities   OK  0.00
test_hosts_up   OK  0.00
test_live_migration_all_checks_pass OK  0.00
test_live_migration_basic   OK  0.00
test_live_migration_compute_dest_not_alive  OK  0.00
test_live_migration_compute_src_not_alive   OK  0.00
test_live_migration_dest_check_service_lack_memory  OK  0.00
test_live_migration_dest_check_service_same_hostOK  0.00
test_live_migration_dest_host_incompatable_cpu_raises   OK  0.00
test_live_migration_dest_hypervisor_version_older_raisesOK  0.00
test_live_migration_different_hypervisor_type_raisesOK  0.00
test_live_migration_different_shared_storage_raises OK  0.00
test_live_migration_instance_not_runningOK  0.00
test_live_migration_same_shared_storage_okayOK  0.00
test_live_migration_volume_node_not_alive   OK  0.00
test_update_service_capabilitiesOK  0.00
CapacityNotifierTestCase
test_bad_event_suffix   OK  0.00
test_bad_publisher_id   OK  0.00
test_event_type OK  0.00
test_update_called  OK  0.00
NotifierListTestCase
test_send_notifications_successfullyOK  0.00
test_send_notifications_with_errors OK  0.00
test_when_driver_fails_to_importOK  0.00

--
Ran 46 tests in 1.986s

OK

Best,
-jay

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] how to run selected tests

2012-02-29 Thread Brad Hall

You can specify directories or files to run_tests.sh (i.e.
./run_tests.sh -N scheduler.test_scheduler notifier to run a subset of
scheduler tests and all of the ones in the notifier directory).

Thanks,
Brad

On Wed, Feb 29, 2012 at 12:42 PM, Yun Mao  wrote:
> Greetings,
>
> What's the most convenient way to run a subset of the existing tests?
> By default run_tests.sh tests everything. For example, I'd like to run
> everything in test_scheduler plus test_notify.py, what's the best way
> to do that? Thanks,
>
> Yun
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] quantum and openvswitch issue

2012-02-29 Thread jay_s_b

Hello,
I have installed one controller node that has 
nova-compute/nova-network/nova-api/nova-volume.
I have installed quantum on another server. I am using openvswitch plugin
 
On the controller node, I have installed openvswitch. I created the 
bridge/port/interface using the following commands
sudo ovs-vsctl add-br br-int
sudo ovs-vsctl set bridge br-int datapath_type=netdev
sudo ovs-vsctl add-port br-int eth1 
 
 I ran the ovs_agent on the controller.
It says, 
INFO:root:Connecting to database "ovs_quantum" on 192.168.198.55
 
But, when  I tried to do
sudo nova-manage --verbose network create --label=public 
--fixed_range_v4=10.0.0.0/24
 
I see network being created in the mysql ovs_quantum database. But, i do not 
see real  network getting created. I do not see any messages in the agent log. 
neither do i see any messages in the quantum-server log.
 
Is there a client to check the communication from ovs_agent on controller to 
quantum-server?
 
Thanks,
-Jay

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] please compress the public nova.git repository (~100MB -> 30MB)

2012-02-29 Thread Jim Meyering

Hello,

Looking at nova.git for the first time, I cloned it.
Surprised that it was so much larger than the others, and too large for
the size of the source and the amount of history, I tried to compress it.
That reduced the repository size from nearly 100MiB to just 30:

  $ git clone git://github.com/openstack/nova.git
  $ git-repo-compress .git
  97M .git
  Counting objects: 101858, done.
  Delta compression using up to 12 threads.
  Compressing objects: 100% (100594/100594), done.
  Writing objects: 100% (101858/101858), done.
  Total 101858 (delta 80708), reused 18388 (delta 0)
  150.91user 81.16system 0:35.05elapsed 662%CPU (0avgtext+0avgdata 
324536maxresident)k
  29056inputs+59312outputs (381major+252932minor)pagefaults 0swaps
  started Tue 2012-02-28 16:40:42 +0100
  Tue 2012-02-28 16:41:17 +0100
  30M .git

The command I used is just a wrapper around git repack:

  git-repo-compress () {
  local d=$1
  du -sh "$d"
  start=$(date)
  env time git --git-dir="$d" repack -afd --window=250 --depth=250
  echo started $start
  date
  du -sh "$d"
  }

Future cloners (and those who push, too) will be
grateful if someone with access to the server would
do the same thing to the public nova.git repository.
Compressing the repo improves server performance, too.

These other repositories compressed well, too:
  size of .git repo (MiB)
   current  compressed
  swift  9.8M2.3M
  keystone11M9.9M
  horizon4.1M3.2M
  glance 5.2M1.9M
  quantum3.0M1.4M

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Memory leaks from greenthreads

2012-02-29 Thread Vishvananda Ishaya

Hello Everyone,

We have had a memory leak due to an interaction with eventlet for a while that 
Johannes has just made a fix for.

bug:
https://bugs.launchpad.net/nova/+bug/903199

fix:
https://bitbucket.org/which_linden/eventlet/pull-request/10/monkey-patch-threadingcurrent_thread-as

Unfortuantely, I don' t think we have a decent workaround for nova while that 
patch is upstreamed.  I wanted to make sure that all of the distros are aware 
of it in case they want to carry an eventlet patch to prevent the slow memory 
leak.

Vish___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] how to run selected tests