I have been implementing some functional stress tests for nova. It
seems that there are various pieces of Python code that have been
written to call the HTTP apis including novaclient as well as code that
is part of various test suites. There are also ueca-tools. It seems like
it would be usef
On 10/17/2011 8:11 AM, Aaron Lee wrote:
It looks like we have two different issues we are trying to solve here. The
first is testing Nova itself, and the second is testing a wrapper to interface
with Nova. I think the wrappers should be tested within their own projects. And
as long as the wrap
th that.
-David
-Original Message-
From: openstack-qa-team-bounces+donald.ngo=hp@lists.launchpad.net
[mailto:openstack-qa-team-bounces+donald.ngo=hp@lists.launchpad.net] On
Behalf Of David Kranz
Sent: Monday, October 17, 2011 9:15 AM
To: openstack-qa-team@lists.launchp
There was a lot of discussion about this in the past month or two but I
am not sure what the conclusion was and I am confused. I think I observe
that:
1. Swift is producing packages for all trunk builds and releases, for
Lucid-Oneiric
2. Nova is not producing packages at all
3. Ubuntu is prod
+kavan.patil=hp@lists.launchpad.net
[mailto:openstack-qa-team-bounces+kavan.patil=hp@lists.launchpad.net] On
Behalf Of David Kranz
Sent: Thursday, November 17, 2011 9:39 PM
To: openstack-qa-team@lists.launchpad.net
Subject: [Openstack-qa-team] Ubuntu packages for milestone releases
The
I tried running Tempest against an existing diablo-stable cluster and
had problems. Is this expected to work?
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
More help
having?
Daryl
On Jan 16, 2012, at 1:44 PM, David Kranz wrote:
I tried running Tempest against an existing diablo-stable cluster and had
problems. Is this expected to work?
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
sion, you could try using 'git bisect'
to identify which commit introduced them.
Thanks,
Mark.
On Mon, 2012-01-16 at 15:01 -0500, David Kranz wrote:
Just some test failures. I will track them down, but first wanted to
make sure Tempest was not intended to be Essex-only.
On 1/16/20
I have been trying to set up to submit changes to tempest but cannot get
past this. I have had a launchpad account for some time and just put in
an ssh key. I followed
the recommended gerrit procedures but in debugging see that the
following command (run with -v) is rejected because the private
On 1/17/2012 3:34 PM, David Kranz wrote:
I have been trying to set up to submit changes to tempest but cannot
get past this. I have had a launchpad account for some time and just
put in an ssh key. I followed
the recommended gerrit procedures but in debugging see that the
following command
With regard to the upcoming goal of having all compute APIs tested
against diablo-stable, it seems that rebuild does not work at all:
https://bugs.launchpad.net/nova/+bug/867627. One of our users reported
this but I didn't see anything about it in the tempest bug list. I think
the just-released
(after long cursing at pep8) I am preparing to check in a set of stress
tests that we have developed and wanted to get some feedback before I
attempt to do so.
The problem to be solved is that nova is a distributed, asynchronous
system that is prone to race condition bugs. These bugs will not
The discussion about Libvirt File Injection on the main list reminded me
that I am not clear on what we mean that a functional test passes. For
example, in one version of the stress test we are going to check in, the
test assigns a floating ip and ssh's to the created server to make sure
it is,
t actions would need to be extended
without seeing the code and understanding what the proposed changes are, but
I'd definitely like to see what you have to figure out how it could integrate
into our work.
Daryl
On Jan 26, 2012, at 11:02 AM, David Kranz wrote:
(after long cursing at pep
Due to some bugs in nova it can happen that a vm fails to start and, due
to another bug, 'nova delete' will not delete it. There was a mention
somewhere that you have to manually remove it from the database but I am
not sure exactly what command will do that. Can some one help me out?
-David
This import occurs at line 41 of tempest/services/image/service.py. I
noticed other places in the code that changed behavior based on whether
glance was there or not. Is this a bug or is glance dependency endorsed?
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to :
Are the tests in Tempest expected to work? I get a lot of failures. Some
tests marked as 'negative' throw exceptions because they try to do
something to a non-existent server. For example:
@attr(type='negative')
def test_delete_nonexistant_server_metadata_item(self):
"""
It seems that a number of Tempest tests are broken because specific
error codes are being checked that are different from those actually being
returned from the API calls. I am puzzled because things like the
floating ip and volume extensions seem to work fine with
nova-client/diablo and
some ha
In tracking down a problem with the tempest flavors test I noticed that
'nova flavor-list' returns this in Essex:
++---+---+--+--+---+-+
| ID |Name | Memory_MB | Swap | Local_GB | VCPUs | RXTX_Factor |
++---+---+--+---
I have noticed that running tempest takes longer than I would like and
seems to spend most of its time waiting for vms to boot. Is there an
image we could use that is known to be the fastest to boot? Should we
(some one who knows how, that is) create one just for this purpose?
-David
--
Mail
There are a bunch of tests that fail in diablo because they expect a
particular 4xx response code and that code is different in diablo and
essex. I have checked in some "fixes" for this by making the test less
rigid about which 4xx code it gets but there are still more I have to
deal with. I a
One failure was a nova bug I reported and just posted this workaround:
https://review.openstack.org/#change,4515.
I reported https://bugs.launchpad.net/tempest/+bug/940532,
test_list_images fails, but am not sure if it is
a tempest or nova bug. The current status is with
resize_available=false.
On 2/24/2012 2:30 PM, Jay Pipes wrote:
Nice job, David! Keep up the good work. I approved that skipper for
the bug so we should be getting close now.
Going to get to the remaining reviews now.
Cheers,
-jay
OK, great. If you create a stable-diablo branch I can get those tests in
shape as we
I am not sure why the resize tests are failing but there is an error in
the compute log so I filed a bug:
https://bugs.launchpad.net/nova/+bug/940619
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://lau
I have filed nova bugs for all the failures in diablo/stable and posted
the changes to make everything pass except for the glance dependency
which is covered by https://bugs.launchpad.net/tempest/+bug/944410. We
decided to make the negative tests (that fail due to a wrong error code)
succeed wi
I am getting the following failure on a new essex cluster using Ubuntu
packages. It seems 404 (NotFound) is expected by Tempest. Has any one
seen this? This test was passing fairly recently...
-David
==
ERROR: Negative test: T
On 3/23/2012 12:18 PM, Jay Pipes wrote:
On 03/23/2012 11:51 AM, David Kranz wrote:
I am getting the following failure on a new essex cluster using Ubuntu
packages. It seems 404 (NotFound) is expected by Tempest. Has any one
seen this? This test was passing fairly recently...
-David
Could it
;status': '400', 'content-length': '70',
'contentype': 'application/json; charset=UTF-8', 'x-compute-request-id':
'req-03ddda01-fdcb-447d-91d4-3b8d4081ac6c'}
tempest.common.rest_client: ERROR: Response Body: {u'badReq
Never mind the last one. That was because I don't have two images. That
test should be skipped in that case.
On 3/23/2012 12:18 PM, Jay Pipes wrote:
On 03/23/2012 11:51 AM, David Kranz wrote:
I am getting the following failure on a new essex cluster using Ubuntu
packages. It seem
I submitted a fix for the markers problem on the essex stable branch and
now all the tests (almost) pass using the real essex release. The only
issue was a non-reproducible failure. I ran this a dozen more times and
did not see it again :-( . So be on the watch for this.
==
We should have as much coverage as possible. But since we have been
using config file variables to control which tests to run based on
configuration (such as resize_available) it would be better to introduce
a new config variable rather than making people tweak the arguments to
nose. I also thi
Does any one have any new items to add to the agenda for this week's
meeting?
So far we have:
1. Getting some consensus on what precisely a "smoke test" is and
discussing some code Jay will have pushed that cleans up our "smoke
test" abilities.
2. Strategy for maintaining the stable/essex t
Is any one close to checking in methods on the volume client that
associates a volume to an instance? I want to use them in a stress test.
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~
The OpenStack QA Team holds public weekly meetings in
#openstack-meeting, Thursday at 13:00 EST (17:00 UTC). Everyone
interested in testing, quality assurance, performance engineering, etc,
should attend!
Agenda for next meeting
* Review of last week's action items
(jaypipes) Get dev-gat
The QA team had its weekly IRC meeting today. Here is a summary of the
progress and decisions coming out of the meeting.
* Progress
There were a bunch of problems that were preventing the tempest gating
job from running successfully. The last (:-) ) problem was identified.
We hope to turn t
On 5/4/2012 9:08 AM, Jay Pipes wrote:
On 05/04/2012 06:14 AM, Karajgi, Rohit wrote:
Hi,
What is the policy that we should or are following for test cases that
fail due to an existing Open Bug in Launchpad?
For eg:
https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floatin
id
On 5/4/2012 9:38 AM, Jay Pipes wrote:
On 05/04/2012 09:28 AM, David Kranz wrote:
On 5/4/2012 9:08 AM, Jay Pipes wrote:
On 05/04/2012 06:14 AM, Karajgi, Rohit wrote:
Hi,
What is the policy that we should or are following for test cases that
fail due to an existing Open Bug in Launchpad?
We were trying to use this feature and could not find such a thing. I
filed a bug against the openstack storage manual for lack of
documentation about this
https://bugs.launchpad.net/openstack-manuals/+bug/994657. The new swift
tempest tests must have to deal with this. How was that handled?
On 5/7/2012 2:29 PM, Venkatesan, Ravikumar wrote:
On 5/4/2012 9:08 AM, Jay Pipes wrote:
On 05/04/2012 06:14 AM, Karajgi, Rohit wrote:
Hi,
What is the policy that we should or are following for test cases that
fail due to an existing Open Bug in Launchpad?
For eg:
https://github.com/openstack/
Can any one say what the current status is of running in parallel reliably?
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
More help : https://help.launchpad.net/ListHe
I will be sending the meeting announcement soon. Please let me know if
you have any agenda items to add to what I have so far:
- Status of Swift tests
- Making parallel tempest runs really work
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team
The weekly QA Team meeting takes place at 17:00 UTC on IRC
(#openstack-meeting on Freenode). We invite anyone interested in
testing, quality assurance and performance engineering to attend the
weekly meeting.
The agenda for this week is as follows:
* Review of last week's action items
- Rav
On 5/30/2012 3:56 PM, Jay Pipes wrote:
> With ubuntu and several other full Linux OSes, the best I've seen
is 15 minute smoke tests and 40-ish minute full regression runs, which
in my case is within my pain threshold. In terms of actions, we've paid
the majority of the time price for testing sce
I am still seeing the below error with the latest versions of
everything. Is this test passing for any of you? It seems to pass in
jenkins so I don't know what the problem could be. Also, that code says:
# Delete the flavor
resp, _ = self.admin_client.delete_flavor(self.new_flav
ng up cached
images on the compute nodes. Dan, do you have any comment about this?
-David
On May 31, 2012, at 3:10 PM, David Kranz wrote:
I am still seeing the below error with the latest versions of everything. Is
this test passing for any of you? It seems to pass in jenkins so I
eally don't understand why the database is never purged of
old stuff since it will just grow and grow. But I am told that is
correct behavior.
-David
On 5/31/2012 4:43 PM, Dan Prince wrote:
- Original Message -
From: "David Kranz"
To: openstack-qa-team@lists.launchpa
@lists.launchpad.net
[mailto:openstack-qa-team-bounces+rohit.karajgi=nttdata@lists.launchpad.net]
*On Behalf Of *Daryl Walleck
*Sent:* Thursday, May 31, 2012 9:52 PM
*To:* David Kranz
*Cc:* openstack-qa-team@lists.launchpad.net
*Subject:* Re: [Openstack-qa-team] Weekly Meeting tomorrow, Thursday,
May 31 @ 17:
I am a little confused about this. Most test classes define
tearDownClass that frees resources allocated in setUpClass. But two of
the classes deviate from this.
ServerActionsTest uses setUp and tearDown and creates a new server in
setUp. I think this means that a new server is created before
) might be different for cases where the test cases take a long
time
c) Consider which, if any, cases it would make sense to use a pool of
pre-allocated servers rather than spinning up new ones for each test.
-David
On Jun 1, 2012, at 4:03 PM, David Kranz wrote:
I am a little confused
There are a few tempest tests, and many in the old kong suite that is
still there, that wait for a server status that is something other than
ACTIVE or VERIFY_RESIZE. These other states, such as BUILD or REBOOT,
are transient so I don't understand why it is correct for code to poll
for those st
On 6/18/2012 1:07 PM, Jay Pipes wrote:
On 06/18/2012 12:49 PM, Daryl Walleck wrote:
I can verify that rescue is a non-race state. The transition is active
to rescue on setting rescue, and rescue to active when leaving rescue.
I don't see a RESCUE state. I see a RESCUED state. Is that what you
.
-David
On 6/18/2012 1:37 PM, Daryl Walleck wrote:
I'm not sure I agree with causing a failure based on what the user inputs to a
function. I feel like this is something we should enforce through standards and
not in code.
Daryl
On Jun 18, 2012, at 12:34 PM, David Kranz wrote:
On 6/18
To help with the effort of making the Tempest suite run faster, we
should avoid or skip the use of soft reboot in any tests, at least for
now. The problem is that, according to Vish, soft reboot requires guest
support. If the booted image doesn't have it, compute will wait (two
minutes by defau
st
hyper visor agnostic, and I would be hesitant to skip a valid test case. I
think it's at least worth noting down as something we can revisit later, but I
think there are other areas we can improve performance in first.
Daryl
Sent from my iPad
On Jun 19, 2012, at 8:01 AM, "David Kranz&qu
Sorry for the late posting. Here are some things to discuss today.
-David
Discuss priorities for Tempest, recorded in
http://etherpad.openstack.org/TempestPriorities
1. Get tempest gating online
Make sure that smoke tests are not duplicated, test basic positive
functionality, and not skip
That's great, Jay. I will remove the skips (assuming they pass :-) ).
-David
On 7/8/2012 11:10 PM, Jay Pipes wrote:
Hey all,
I went ahead and created a script that you can run to see the status of
any Launchpad bugs that are causing Tempest tests to be skipped:
https://review.openstack.org/#
I just submitted a removal of the skips. Note that there were two user
role tests that still failed in a way I could not decipher so I filed a
tempest bug https://bugs.launchpad.net/tempest/+bug/1022990.
Unfortunately, when the skip script went in with the below checkin
message, it caused Jenk
I have been triaging tempest bugs and am not sure what to do about
these. Several had patches submitted which expired. Does any one care
enough about these to do the work and test it, or should these bugs be
closed? These bugs are assigned right now to Sarad Patel.
-David
--
Mailing list: h
Folks, I noticed that there a number of bugs that are assigned and are
In Progress where it is not clear whether the assignee is actually
working on them. If you are the assignee of a bug In Progress and do not
expect to submit a fix in the next few weeks, please change the status
to confirmed.
There are a number of bug tickets that are assigned and marked as
Triaged or Confirmed that have not been touched in a long time. Please
look over any bugs in such a state that are assigned to you and change
the owner to nobody if you do not think (be honest!) fixing them will
ever make it to
I think this clearly illustrates why it is so important for QA to be
done more in the open, and to post tempest blueprints for largish
projects like this. I have a theory that one important reason so many
organizations choose not to do this is that once they submit their tests
to the public rep
gt; folks I can reach out to help stabilize the environment.
Yeah, where is that slacker? :)
We're very appreciative of the review you've already provided, as well
as your efforts to help move those forward. I didn't intend to come off
whiny, I just wanted to make sure that we don&
how
to do that.
-David
On 8/22/2012 3:15 PM, James E. Blair wrote:
David Kranz writes:
I just submitted https://review.openstack.org/#/c/11755/ to skip the
tests that are failing. I filed three bug tickets too. I can only
reproduce one of them in my environment (the one where the error cod
On 8/29/2012 8:41 AM, Jaroslav Henner wrote:
Hi.
I made my first little patch:
https://review.openstack.org/#/c/12048/
which failed 3 times on
http://logs.openstack.org/12048/1/check/gate-tempest-devstack-vm/9570
which surely is not my issue. What can I do with this? Also, who
should I ask to
In case any one else didn't know this, I just found out that a "recheck"
comment reruns the build but "reverify" will also commit
the change if the necessary code review +2 and approved +1 are already
present.
There are a bunch of remaining flaky test failures that need to be
diagnosed. I am w
Thanks, Jim. That sounds like a good start. One thing that concerned me
was that some of the tests in tempest that I had to skip to get tempest
working were flakey. There are still some existing flakeys. It is not
clear what part of this is due to jenkins infrastructure vs nova
regressions or o
A bunch of tests are failing now. I can reproduce some of them locally
and filed https://bugs.launchpad.net/nova/+bug/1046870.
https://bugs.launchpad.net/tempest/+bug/1039739 may also be related to this.
This situation is pretty bad.
-David
On 9/5/2012 9:03 PM, James E. Blair wrote:
Hi,
In
Jim, we discussed this at the QA meeting and would appreciate your going
ahead with this.
We are going to raise the current problems at the nova meeting today and
see if we can get more attention to the
breakages whose tests in tempest have been commented out. We hope that
(3), because it dire
Do we have a policy about whether bug tickets are needed for every
change? I happened to see a silly coding error and would prefer to avoid
the overhead of a bug ticket for such things.
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.laun
As of a few hours ago the tempest gate is unblocked. However, it seems
that all the pending changes need to be rebased.
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
Mo
I heard from some of my team members that test_server_basic_ops and
test_server_advanced_ops were failing and I can reproduce it with
current devstack/tempest.
Looking at the code it seems that the keystone Client object does not
have a service_catalog object like the error says. So why is this
On 9/25/2012 10:35 AM, Jay Pipes wrote:
On 09/25/2012 09:38 AM, David Kranz wrote:
I heard from some of my team members that test_server_basic_ops and
test_server_advanced_ops were failing and I can reproduce it with
current devstack/tempest.
Looking at the code it seems that the keystone
edentials in the code
just below.
-David
On Tue, Sep 25, 2012 at 9:35 AM, Jay Pipes <mailto:jaypi...@gmail.com>> wrote:
On 09/25/2012 09:38 AM, David Kranz wrote:
> I heard from some of my team members that test_server_basic_ops and
> test_server_advanced_op
Folks, there are already a number of proposals for sessions that can be
seen at http://summit.openstack.org/. I will be reviewing them early
next week so I encourage any one who wants to lead a session, or has a
topic that should be discussed, to make a proposal at that page. If you
want to pr
1. Flaky failing volume tests
2. Parallel tempest tests
3. Design summit sessions
4. Status of swift tests
5. Is XML done?
6. Status of quantum tests
Any one have anything else?
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.ne
We discussed this a bit at the meeting today. Monty has proposed a
session on the QA track about parallelizing some of the CI stuff. He
believes tempest could share the parallelization code. See
http://summit.openstack.org/cfp/details/69.
Parallelizing the tempest gate job is as much of a ci iss
ssues we're trying to address.
Daryl
From: openstack-qa-team-bounces+daryl.walleck=rackspace@lists.launchpad.net
[openstack-qa-team-bounces+daryl.walleck=rackspace@lists.launchpad.net] on
behalf of David Kranz [david.kr...@qrclab.com]
Sent
25, 2012 at 9:35 AM, Jay Pipes <mailto:jaypi...@gmail.com>> wrote:
On 09/25/2012 09:38 AM, David Kranz wrote:
> I heard from some of my team members that test_server_basic_ops and
> test_server_advanced_ops were failing and I can reproduce it with
> current devs
and merged.
On 09/28/2012 11:51 AM, David Kranz wrote:
This was the problem (trivial) https://review.openstack.org/#/c/13840/.
Some one please review.
I am not sure when the behavior changed.
-David
On 9/25/2012 10:59 AM, Dolph Mathews wrote:
That generally pops up when you're bypassing
Pipes wrote:
That's because I forgot to decorate the base SmokeTest with the
@attr(type="smoke") decorator :(
Feel like doing that?
-jay
On 09/28/2012 01:05 PM, David Kranz wrote:
Thanks, Jay. But this now confirms that test_server_basic_ops is not
running in the gating job. But it
It was recently discovered that the gating job was not running tests
with no @attr. One of the tests that was not being run as a result of
this is broken in at least its XML component. It would be great if one
of the folks who worked on the XML stuff could pick this up soon:
https://bugs.launch
As of late yesterday, the full tempest gate is running all tempest
tests. Not surprisingly, there are some failures in the tests that have
just started running. Most of the problems seem to be due to some recent
change
in the keystone client but there may be others. We are working to get it
bac
A brief status update:
1. The fixes for the full tempest gate have been merged. Tempest should
run without errors except
that there is still a flakey volume test
https://bugs.launchpad.net/tempest/+bug/1056213.
This is not a bug in tempest and we are struggling with what to do
about it
On 10/5/2012 3:16 PM, Yaniv Kaul wrote:
On 10/05/2012 07:50 PM, David Kranz wrote:
A brief status update:
1. The fixes for the full tempest gate have been merged. Tempest
should run without errors except
that there is still a flakey volume test
https://bugs.launchpad.net/tempest/+bug
On 10/25/2012 1:13 AM, Daryl Walleck wrote:
While spinning up a new devstack tonight I noticed some very odd
behavior. Keystone is suddenly giving me back a 3000+ character auth
token, and the ids for flavors I'm creating are extremely large ints
(uuids I could see, but not this). Does anyone h
Going forward, such changes should really be announced to the dev list.
There is no reason people need to be
left to trip over them.
-David
On 10/25/2012 1:13 PM, Adam Young wrote:
On 10/25/2012 12:49 PM, Daryl Walleck wrote:
You hit the nail on the head. I got a bit jumpy and ended up filing
There is now a full tempest run going daily and reporting failures to
this list. But that won't work because
jenkins and gerrit cannot be launchpad members. According to the ci
folks, others have dealt with this
by moving their mailing lists to lists.openstack.org. Perhaps we should
do the same?
We created a new mailing list (openstack...@list.openstack.org) that is
controlled by openstack and can receive mail from automated builds and such.
Unfortunately there is no real way to automatically migrate users to the
new list. So you need to register for the new list at
http://lists.opens
We will all start forgetting about the old list so please do this if you
are still interested in the qa efforts.
-David
Original Message
Subject:[openstack-qa] Proposed QA Team meeting agenda for today
Date: Thu, 08 Nov 2012 09:45:30 -0500
From: Sean Dague
Reply-
And just a reminder to those in the US (at least) that daylight savings
time starts this weekend so the meeting will start one hour later
(1pmEDT) than it has been for the last few months.
-David
On 3/8/2013 12:56 PM, Jay Pipes wrote:
Hello QAers,
Just a quick note to say next week's IRC me
90 matches
Mail list logo
