[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[remibergsma]

Github user remibergsma commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-151455578
  
@pvr9711 There are so many things to do for 4.6, do you really want _me_ to 
send a PR for moving a file? Please send it yourself, and both @miguelaferreira 
and me will OK it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[runseb]

Github user runseb commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-151584850
  
@nitt10prashant @pvr9711 please close this thread I will take care of 
moving the file


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[nitt10prashant]

Github user nitt10prashant commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-151720250
  
based on discussion closing the PR 
Thanks @remibergsma @runseb @miguelaferreira @pvr9711  for great discussion 
.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[nitt10prashant]

Github user nitt10prashant closed the pull request at:

https://github.com/apache/cloudstack/pull/960


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[pvr9711]

Github user pvr9711 commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-151071811
  
+1 move this test out of BVTs - this was brought up on almost 3-4 weeks 
ago. 
@runseb - I guess, my understanding is - we need 100% passrate on all BVTs 
- we cannot have any skipped or failed.  In this case not everyone will have a 
nicira appliance and be able to test it.  Hence it fails to be a common BVT - 
it can be a BVT for someone who has a nicira appliance.  That said, If someone 
has any specific test(s) that only they can run - keep them in a separate 
folder.   This was discussed in an email chain almost 4 weeks ago and we left 
it open. well enough said on this, :-) 

@remibergsma   @miguelaferreira can you please take care of moving this 
file ?  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[remibergsma]

Github user remibergsma commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150535530
  
Who's going to propose a PR that moves the test? Then we're done and can 
focus on getting 4.6 released!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[nitt10prashant]

Github user nitt10prashant commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150501158
  
@runseb
"I personally fail to see the benefit of skipped tests versus a failed test"
If i may ,
skip  is a type of failure where you know in advance  about failure  , 
since you know about the failure you just Handle them  in order to 
differentiate from a general failure(may be product issue).So its all about 
categorizing  the failure  like expected failure and unexpected failure 

Its about how you see reports 
If you accept skip as known failure then 100% success on BVT ~ 20% of 
skipped =80% success with 20% of known failure ,at least not a product issue 
which you can say  without looking at particular script .
Without skip  you have to analyze each and every failure then you may come 
up with % of product  issue and % of failure due to setup an all

"While @nitt10prashant and @pvr9711 you seem to want 100% success on BVT 
even if it includes some tests that are actually not run."
My comment:
1-Not run test cases are  the  one which are not picked up by runner may be 
coz of different tags and those test cases will not appear  in your report 
2-skipped test case are the one which are picked up by runner but due to 
non availability of configurations or some other reasons  could not run 
completely  and they will appear in your report .
3-I want 100% success on BVT  without any skip  if all the setup config is 
available .incase they are not i would prefer to skip them in order to find out 
real failure  which does not include any failure because of setup/configuration 
 just by looking at my report .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[pvr9711]

Github user pvr9711 commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150153497
  
LGTM - skipping testing is designed with a purpose and this is exactly we 
should be using it for.  As a as person running/maintaining the tests, someone 
would have to look at all tests- passed or failed or skipped.  The message for 
skipping should provide someone whos running the tests whether the 
configuration is not there or incorrect or there is something else to it.  
@runseb - request you to chime in and would like to avoid discussions on 
this.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[miguelaferreira]

Github user miguelaferreira commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150141365
  
We have had a lengthy discussion about this on the mailing list. I didn't' 
agree that skipping tests is better than excluding them from a test run.

I rather have a broken test run because I forgot to configure it properly, 
than a successful test run where a test didn't run (and I forgot, same issue, 
to double check).

There are too many situations where the ACS builds succeed because test are 
being silently skipped, and people don't double check.

:-1: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[runseb]

Github user runseb commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150178317
  
I think you guys agree on the problem but disagree on the solution.
To me 100% success on BVT, if it includes ~20% (or whatever percentage) of 
skipped tests is not a real 100% success rate on BVT.

So @miguelaferreira prefers to see 80% success on BVT and vote +1 on a 
release, if he actually knows what's happening with the 20% failure.
While @nitt10prashant and @pvr9711 you seem to want 100% success on BVT 
even if it includes some tests that are actually not run.

The nicira test is actually the only one in there that's testing a third 
party plugin. I don't see any tests for solidfire, netscaler, open contrail, 
midokura etc….All theses tests will fail without the proper hardware/software 
setup and hence will fail with simulator run.

If we really want 100% on integration tests (so called BVT) we need to 
define a test suite that everyone can run without special hardware or software. 
And those tests *cannot* be skipped.

I personally fail to see the benefit of skipped tests versus a failed test.

For this particular tests I propose we move it to a folder 
/test/integration/plugins and have our base tests defining "BVT" as the test in 
/test/integration/smoke

So -1 on this PR
+1 on a PR that would move this test to a separate folder.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[miguelaferreira]

Github user miguelaferreira commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150185163
  
The directory in which a test lives is irrelevant to me, so I would also +1 
a PR that moves this to another directory.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[pvr9711]

Github user pvr9711 commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150168452
  
@miguelaferreira - we have discussed this on an earlier email.  as Prashant 
said, In this case a test case is skipped because the setup does not have the 
configuration needed for the test to run.  the test will fail if the 
configuration is incorrect.  From that sense, IMHO - test setup method is not 
coded to handle the possibility of configuration not being present and hence 
needs a fix.   

Just my 2 cents,  I would like to skip a test if I don't have the 
configuration.  As a person running the BVTs, I will check all the test status 
and cannot ignore skipped tests.   Also, BVTs are run every time when the build 
is kicked off.  If people keep adding their own tests which not everyone can 
run then the importance of BVTs is gone and we will never achieve 100% 
passrate.  If someone has specific tests they want to run, keep it in a 
different folder and not keep it in smoke folder. Also, skipping tests is 
designed for a purpose.  

@runseb we should establish some ground rules around BVTs otherwise we will 
have to do conditional coding for running tests which is hard to maintain.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[miguelaferreira]

Github user miguelaferreira commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150169879
  
Does 100% pass rate when some of the tests have been skipped mean more than 
100% pass rate when the same tests have been excluded?

I only hear about these BTV tests when a release is being prepared (I could 
be wrong, and you maybe run these tests all the time). I'm actually building a 
CI setup that will run every day all day, for all PRs. In such a setup I would 
have made my live easier by just taking all the test available. But instead I 
choose to careful select the test I'm running, to be sure about what I'm 
testing. I already see that several tests need to be improved, and then more 
tests added, and so on. This will lead to sustainable improvement.

Now, if you say that it is hard to maintain a list of tests to include in a 
build, I would say that is the problem of the build and not the tests.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[pvr9711]

Github user pvr9711 commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150171997
  
@miguelaferreira 100% passrate on the BVTs - if there are skipped cases you 
should/want to find out why they are getting skipped and fix them.  If you are 
skipping tests always they should not be in the list of BVTs.  we cannot skip 
all the tests and declare victory!

We also have Regression Tests - in the integration/component folder - which 
is the next area that we should go after and get a high passrate like 97-98, if 
not 100% because getting 100% can be problem in this area due to known issues 
that may exist ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[nitt10prashant]

Github user nitt10prashant commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150161922
  
@miguelaferreira  i  was going though mail thread and found this 
"I think the test should fail if the config is not right, or otherwise be 
excluded if the indentation is to not run them." 
My comment 
1-For not right nicira settings , test case will fail
2-It will skip only if nicira settings are not present in config file

skipping test case or excluding it from a run ?
I would prefer skipping based on config file with proper message in test 
results .
To exclude  test file  you  have to  modify the job which will be painful 
if i have to do it for a number of test scripts .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[miguelaferreira]

Github user miguelaferreira commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150162817
  
@nitt10prashant I rather not skip at all. There are just too many examples 
of tests being skipped and people not reviewing the output. And that's not even 
the worse case scenario! This week I made a PR, I got ! LGTM, Jenkins was 
green, travis was green and maven wouldn't even compile the project because I 
broke it. Builds and test runs breaking when something is not good is what we 
need.

And in all fairness, this whole discussion is about not having to give any 
thought to which test will run in someone's builds. Why go through all this 
trouble just to be able to run all tests in the `test/integration` directory? 
Will that make ACS better?
Take the time to figure out which tests you want to run, and if you don't 
want to run this one, just don't include it in you build!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[remibergsma]

Github user remibergsma commented on the pull request:

https://github.com/apache/cloudstack/pull/960#issuecomment-150032083
  
pinging @miguelaferreira 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] cloudstack pull request: CLOUDSTACK-8975:skip test case if niciraN...

[nitt10prashant]

GitHub user nitt10prashant opened a pull request:

https://github.com/apache/cloudstack/pull/960

CLOUDSTACK-8975:skip test case if niciraNvp settings are not available in 
config file

root/cloudstack/test/integration/smoke/test_nicira_controller.py is failing 
if niciraNvp settings are not available in config file. Modifying script to 
skip test case if settings are not available 

test result
===
test_01_nicira_controller 
(integration.smoke.test_nicira_controller.TestNiciraContoller) ... SKIP: nicira 
settings not found in config file
Nicira clusters will redirect clients (in this case ACS) to the master 
node. ... SKIP: nicira settings not found in config file

--
Ran 2 tests in 0.001s

OK (SKIP=2)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nitt10prashant/cloudstack nicira

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/cloudstack/pull/960.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #960


commit 07b32dfa61cdaa5ed7140ce577cce2fdcfaf79e0
Author: nitt10prashant 
Date:   2015-10-21T10:39:47Z

CLOUDSTACK-8975:skip test case if niciraNvp settings are not available in 
config file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---