Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Thanks Salvatore. We will try to rebase the patches with master in our environment before running devstack. From: Salvatore Orlando [mailto:sorla...@nicira.com] Sent: Tuesday, August 26, 2014 3:53 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Hi Karthik, what do you mean that the plugin is incompatible with https://review.openstack.org/#/c/114393/? you're mentioning a rebase issue - but the patch in question appears to cleanly apply to master. Is your probably because patch #114393 does not have in its log some changes you need to accommodate test_lib changes? Are these changes you need already merged? In this case you might try to rebase the patch you're going to test on master before running devstack, which I think it's also what happens in the upstream gate. Salvatore On 26 August 2014 21:57, Karthik Natarajan mailto:natar...@brocade.com>> wrote: Hi Edgar, We are also facing CI issues when the neutron patch set is not rebased with latest changes. For e.g. CI audit that you posted today (https://review.openstack.org/#/c/114393/) is not rebased with neutron test_lib related changes. We had refactored the Brocade Vyatta plugin unit tests to accommodate the test_lib related changes. But our plugin is not compatible with the patch you have posted. So CI is failing. I had a discussion with Dane Leblanc on this. We also need to post the SKIPPED status for such patch sets. We will also experiment with Kevin's suggestion. Thanks, Karthik -Original Message- From: Dane Leblanc (leblancd) [mailto:lebla...@cisco.com<mailto:lebla...@cisco.com>] Sent: Monday, August 25, 2014 10:02 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar, Kyle: Kevin's suggestion should work for me (still hashing out the implementation). I've added an item to the 3rd Party IRC agenda anyway to discuss this corner case. Thanks! Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com<mailto:edgar.mag...@workday.com>] Sent: Monday, August 25, 2014 12:44 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Dane, I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you. Thanks, Edgar On 8/25/14, 10:08 AM, "Kyle Mestery" mailto:mest...@mestery.com>> wrote: >Dane, thanks for all the great work you're doing in the third-party CI >area. It's great to see you working to share this knowledge with others >as well! > >Did Kevin's idea work for you to move past this issue? If not, I >suggest you put an item on the neutron meeting agenda today and we >cover this there. You could put the item on the third-party meeting >agenda as well. > >Thanks! >Kyle > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) >mailto:lebla...@cisco.com>> wrote: >> Hi Kevin: >> >> >> >> Thanks, this is a great idea! I may try just a slight variation of >>this concept. Maybe your idea could be the recommended way to create >>a 3rd party CI for plugins that are just being introduced and need to >>limit the scope of testing to a small set of plugin-related commits >>(or plugins blocked on a certain fix). >> >> >> >> Thanks, >> Dane >> >> >> >> From: Kevin Benton [mailto:blak...@gmail.com<mailto:blak...@gmail.com>] >> Sent: Saturday, August 23, 2014 5:47 AM >> >> >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >> >> >> Can you disable posting of results directly from your Jenkins/Zuul >>setup and have a script that just checks the log file for special >>markers to determine if the vote should be FAILED/PASSED/SKIPPED? >>Another advantage of this approach is that it gives you an >>opportunity to detect when a job just failed to setup due to >>infrastructure reasons and trigger a recheck without ever first >>posting a failure to gerrit. >> >> >> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) >> mailto:lebla...@cisco.com>> wrote: >> >> Thanks Edgar for updating the APIC status!!! >> >> Edgar and Kyle: *PLEASE NOTE** I need your understanding >>and advice on the following: >> >> We are still stuck with a problem stemming from a design limitation >>
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
On 2014-08-27 00:52:35 +0200 (+0200), Salvatore Orlando wrote: [...] > In this case you might try to rebase the patch you're going to > test on master before running devstack, which I think it's also > what happens in the upstream gate. More or less. Technically we try to merge the proposed change (and its open parent changes in Gerrit if any, in series) onto the current target branch tip. Rebasing/cherry-picking are a little muddier since they alter the commits being tested rather than merely adding merge commits to the mix. Also, upstream's Zuul (the "Jenkins" account in Gerrit) now automatically runs a built-in job against all open changes for a project+branch every time a change is merged for that combination, testing to see if the remaining open changes can still merge without conflict and leaves a -1 verify on them with a relevant message if not. So if a third-party CI is encountering changes which really can't be merged (rebased, cherry-picked, whatever) to the project then they should refrain from commenting at all... at this point it would be at best redundant, and at worst entirely incorrect. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Karthik, what do you mean that the plugin is incompatible with https://review.openstack.org/#/c/114393/? you're mentioning a rebase issue - but the patch in question appears to cleanly apply to master. Is your probably because patch #114393 does not have in its log some changes you need to accommodate test_lib changes? Are these changes you need already merged? In this case you might try to rebase the patch you're going to test on master before running devstack, which I think it's also what happens in the upstream gate. Salvatore On 26 August 2014 21:57, Karthik Natarajan wrote: > Hi Edgar, > > We are also facing CI issues when the neutron patch set is not rebased > with latest changes. > For e.g. CI audit that you posted today ( > https://review.openstack.org/#/c/114393/) is not rebased with neutron > test_lib related changes. > We had refactored the Brocade Vyatta plugin unit tests to accommodate the > test_lib related changes. > But our plugin is not compatible with the patch you have posted. So CI is > failing. > > I had a discussion with Dane Leblanc on this. We also need to post the > SKIPPED status for such patch sets. > We will also experiment with Kevin's suggestion. > > Thanks, > Karthik > > -Original Message- > From: Dane Leblanc (leblancd) [mailto:lebla...@cisco.com] > Sent: Monday, August 25, 2014 10:02 AM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > Edgar, Kyle: > > Kevin's suggestion should work for me (still hashing out the > implementation). I've added an item to the 3rd Party IRC agenda anyway to > discuss this corner case. > > Thanks! > Dane > > -Original Message- > From: Edgar Magana [mailto:edgar.mag...@workday.com] > Sent: Monday, August 25, 2014 12:44 PM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > Dane, > > I will second Kyle's idea. Let's discuss this during today IRC meeting if > Kevin's suggestion does not work for you. > > Thanks, > > Edgar > > On 8/25/14, 10:08 AM, "Kyle Mestery" wrote: > > >Dane, thanks for all the great work you're doing in the third-party CI > >area. It's great to see you working to share this knowledge with others > >as well! > > > >Did Kevin's idea work for you to move past this issue? If not, I > >suggest you put an item on the neutron meeting agenda today and we > >cover this there. You could put the item on the third-party meeting > >agenda as well. > > > >Thanks! > >Kyle > > > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) > > wrote: > >> Hi Kevin: > >> > >> > >> > >> Thanks, this is a great idea! I may try just a slight variation of > >>this concept. Maybe your idea could be the recommended way to create > >>a 3rd party CI for plugins that are just being introduced and need to > >>limit the scope of testing to a small set of plugin-related commits > >>(or plugins blocked on a certain fix). > >> > >> > >> > >> Thanks, > >> Dane > >> > >> > >> > >> From: Kevin Benton [mailto:blak...@gmail.com] > >> Sent: Saturday, August 23, 2014 5:47 AM > >> > >> > >> To: OpenStack Development Mailing List (not for usage questions) > >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > >>required to be run > >> > >> > >> > >> Can you disable posting of results directly from your Jenkins/Zuul > >>setup and have a script that just checks the log file for special > >>markers to determine if the vote should be FAILED/PASSED/SKIPPED? > >>Another advantage of this approach is that it gives you an > >>opportunity to detect when a job just failed to setup due to > >>infrastructure reasons and trigger a recheck without ever first > >>posting a failure to gerrit. > >> > >> > >> > >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) > >> wrote: > >> > >> Thanks Edgar for updating the APIC status!!! > >> > >> Edgar and Kyle: *PLEASE NOTE** I need your understanding > >>and advice on the following: > >> > >> We are still stuck with a problem stemming from a design limitation > >> of Jenkins that prevents us from being compliant with N
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, We are also facing CI issues when the neutron patch set is not rebased with latest changes. For e.g. CI audit that you posted today (https://review.openstack.org/#/c/114393/) is not rebased with neutron test_lib related changes. We had refactored the Brocade Vyatta plugin unit tests to accommodate the test_lib related changes. But our plugin is not compatible with the patch you have posted. So CI is failing. I had a discussion with Dane Leblanc on this. We also need to post the SKIPPED status for such patch sets. We will also experiment with Kevin's suggestion. Thanks, Karthik -Original Message- From: Dane Leblanc (leblancd) [mailto:lebla...@cisco.com] Sent: Monday, August 25, 2014 10:02 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar, Kyle: Kevin's suggestion should work for me (still hashing out the implementation). I've added an item to the 3rd Party IRC agenda anyway to discuss this corner case. Thanks! Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Monday, August 25, 2014 12:44 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Dane, I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you. Thanks, Edgar On 8/25/14, 10:08 AM, "Kyle Mestery" wrote: >Dane, thanks for all the great work you're doing in the third-party CI >area. It's great to see you working to share this knowledge with others >as well! > >Did Kevin's idea work for you to move past this issue? If not, I >suggest you put an item on the neutron meeting agenda today and we >cover this there. You could put the item on the third-party meeting >agenda as well. > >Thanks! >Kyle > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) > wrote: >> Hi Kevin: >> >> >> >> Thanks, this is a great idea! I may try just a slight variation of >>this concept. Maybe your idea could be the recommended way to create >>a 3rd party CI for plugins that are just being introduced and need to >>limit the scope of testing to a small set of plugin-related commits >>(or plugins blocked on a certain fix). >> >> >> >> Thanks, >> Dane >> >> >> >> From: Kevin Benton [mailto:blak...@gmail.com] >> Sent: Saturday, August 23, 2014 5:47 AM >> >> >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >> >> >> Can you disable posting of results directly from your Jenkins/Zuul >>setup and have a script that just checks the log file for special >>markers to determine if the vote should be FAILED/PASSED/SKIPPED? >>Another advantage of this approach is that it gives you an >>opportunity to detect when a job just failed to setup due to >>infrastructure reasons and trigger a recheck without ever first >>posting a failure to gerrit. >> >> >> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) >> wrote: >> >> Thanks Edgar for updating the APIC status!!! >> >> Edgar and Kyle: *PLEASE NOTE** I need your understanding >>and advice on the following: >> >> We are still stuck with a problem stemming from a design limitation >> of Jenkins that prevents us from being compliant with Neutron 3rd >> Party CI requirements for our DFA CI. >> >> The issue is that Jenkins only allows our scripts to >>(programmatically) return either Success or Fail. There is no option >>to return "Aborted", "Not Tested", or "Skipped". >> >> Why does this matter? The DFA plugin is just being introduced, and >>initial DFA-enabling change sets have not yet been merged. Therefore, >>all other change sets will fail our Tempest tests, since they are not >>DFA-enabled. >> >> Similarly, we were recently blocked in our APIC CI with a critical >> bug, causing all change sets without this fix to fail on our APIC testbed. >> >> In these cases, we would like to enter a "throttled" or "partially >>blocked" >> mode, where we would skip testing on change sets we know will fail, >>and (in an ideal world) signal this shortcoming to Gerrit e.g. by >>returning a "Skipped" status. Unfortunately, this option is not >>available in Jenkins scripts, as Jenkins is currently des
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Edgar, Kyle: Kevin's suggestion should work for me (still hashing out the implementation). I've added an item to the 3rd Party IRC agenda anyway to discuss this corner case. Thanks! Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Monday, August 25, 2014 12:44 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Dane, I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you. Thanks, Edgar On 8/25/14, 10:08 AM, "Kyle Mestery" wrote: >Dane, thanks for all the great work you're doing in the third-party CI >area. It's great to see you working to share this knowledge with others >as well! > >Did Kevin's idea work for you to move past this issue? If not, I >suggest you put an item on the neutron meeting agenda today and we >cover this there. You could put the item on the third-party meeting >agenda as well. > >Thanks! >Kyle > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) > wrote: >> Hi Kevin: >> >> >> >> Thanks, this is a great idea! I may try just a slight variation of >>this concept. Maybe your idea could be the recommended way to create >>a 3rd party CI for plugins that are just being introduced and need to >>limit the scope of testing to a small set of plugin-related commits >>(or plugins blocked on a certain fix). >> >> >> >> Thanks, >> Dane >> >> >> >> From: Kevin Benton [mailto:blak...@gmail.com] >> Sent: Saturday, August 23, 2014 5:47 AM >> >> >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >> >> >> Can you disable posting of results directly from your Jenkins/Zuul >>setup and have a script that just checks the log file for special >>markers to determine if the vote should be FAILED/PASSED/SKIPPED? >>Another advantage of this approach is that it gives you an >>opportunity to detect when a job just failed to setup due to >>infrastructure reasons and trigger a recheck without ever first >>posting a failure to gerrit. >> >> >> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) >> wrote: >> >> Thanks Edgar for updating the APIC status!!! >> >> Edgar and Kyle: *PLEASE NOTE** I need your understanding >>and advice on the following: >> >> We are still stuck with a problem stemming from a design limitation >> of Jenkins that prevents us from being compliant with Neutron 3rd >> Party CI requirements for our DFA CI. >> >> The issue is that Jenkins only allows our scripts to >>(programmatically) return either Success or Fail. There is no option >>to return "Aborted", "Not Tested", or "Skipped". >> >> Why does this matter? The DFA plugin is just being introduced, and >>initial DFA-enabling change sets have not yet been merged. Therefore, >>all other change sets will fail our Tempest tests, since they are not >>DFA-enabled. >> >> Similarly, we were recently blocked in our APIC CI with a critical >> bug, causing all change sets without this fix to fail on our APIC testbed. >> >> In these cases, we would like to enter a "throttled" or "partially >>blocked" >> mode, where we would skip testing on change sets we know will fail, >>and (in an ideal world) signal this shortcoming to Gerrit e.g. by >>returning a "Skipped" status. Unfortunately, this option is not >>available in Jenkins scripts, as Jenkins is currently designed. The >>only options we have available is "Success" or all "Fail", which are >>both misleading. We would also incorrectly report success or fail on >>one of the following test >> commits: >> https://review.openstack.org/#/c/114393/ >> https://review.openstack.org/#/c/40296/ >> >> I've brought this issue up on the openstack-infra IRC, and jeblair >>confirmed the Jenkins limitation, but asked me to get consensus from >>the Neutron community as to this being a problem/requirement. I've >>also sent out an e-mail on the Neutron ML trying to start a >>discussion on this problem (no traction). I plan on bringing this up >>in the 3rd Party CI IRC on Monday, assuming there is time permitted >>in the open discussion. >> >
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Dane, I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you. Thanks, Edgar On 8/25/14, 10:08 AM, "Kyle Mestery" wrote: >Dane, thanks for all the great work you're doing in the third-party CI >area. It's great to see you working to share this knowledge with >others as well! > >Did Kevin's idea work for you to move past this issue? If not, I >suggest you put an item on the neutron meeting agenda today and we >cover this there. You could put the item on the third-party meeting >agenda as well. > >Thanks! >Kyle > >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) > wrote: >> Hi Kevin: >> >> >> >> Thanks, this is a great idea! I may try just a slight variation of this >> concept. Maybe your idea could be the recommended way to create a 3rd >>party >> CI for plugins that are just being introduced and need to limit the >>scope of >> testing to a small set of plugin-related commits (or plugins blocked on >>a >> certain fix). >> >> >> >> Thanks, >> Dane >> >> >> >> From: Kevin Benton [mailto:blak...@gmail.com] >> Sent: Saturday, August 23, 2014 5:47 AM >> >> >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required >> to be run >> >> >> >> Can you disable posting of results directly from your Jenkins/Zuul >>setup and >> have a script that just checks the log file for special markers to >>determine >> if the vote should be FAILED/PASSED/SKIPPED? Another advantage of this >> approach is that it gives you an opportunity to detect when a job just >> failed to setup due to infrastructure reasons and trigger a recheck >>without >> ever first posting a failure to gerrit. >> >> >> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) >> wrote: >> >> Thanks Edgar for updating the APIC status!!! >> >> Edgar and Kyle: *PLEASE NOTE** I need your understanding >>and >> advice on the following: >> >> We are still stuck with a problem stemming from a design limitation of >> Jenkins that prevents us from being compliant with Neutron 3rd Party CI >> requirements for our DFA CI. >> >> The issue is that Jenkins only allows our scripts to (programmatically) >> return either Success or Fail. There is no option to return "Aborted", >>"Not >> Tested", or "Skipped". >> >> Why does this matter? The DFA plugin is just being introduced, and >>initial >> DFA-enabling change sets have not yet been merged. Therefore, all other >> change sets will fail our Tempest tests, since they are not DFA-enabled. >> >> Similarly, we were recently blocked in our APIC CI with a critical bug, >> causing all change sets without this fix to fail on our APIC testbed. >> >> In these cases, we would like to enter a "throttled" or "partially >>blocked" >> mode, where we would skip testing on change sets we know will fail, and >>(in >> an ideal world) signal this shortcoming to Gerrit e.g. by returning a >> "Skipped" status. Unfortunately, this option is not available in Jenkins >> scripts, as Jenkins is currently designed. The only options we have >> available is "Success" or all "Fail", which are both misleading. We >>would >> also incorrectly report success or fail on one of the following test >> commits: >> https://review.openstack.org/#/c/114393/ >> https://review.openstack.org/#/c/40296/ >> >> I've brought this issue up on the openstack-infra IRC, and jeblair >>confirmed >> the Jenkins limitation, but asked me to get consensus from the Neutron >> community as to this being a problem/requirement. I've also sent out an >> e-mail on the Neutron ML trying to start a discussion on this problem >>(no >> traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, >> assuming there is time permitted in the open discussion. >> >> I'm also investigating >> >> For the short term, I would like to propose the following: >> * We bring this up on the 3rd Party CI IRC on Monday to get a solution >>or >> workaround, if available. If a solution is available, let's consider >> including that as a hint when we come up with CI requirements for >>handling >> CIs bocked by some critical fix. >> * I'm al
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Dane, thanks for all the great work you're doing in the third-party CI area. It's great to see you working to share this knowledge with others as well! Did Kevin's idea work for you to move past this issue? If not, I suggest you put an item on the neutron meeting agenda today and we cover this there. You could put the item on the third-party meeting agenda as well. Thanks! Kyle On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd) wrote: > Hi Kevin: > > > > Thanks, this is a great idea! I may try just a slight variation of this > concept. Maybe your idea could be the recommended way to create a 3rd party > CI for plugins that are just being introduced and need to limit the scope of > testing to a small set of plugin-related commits (or plugins blocked on a > certain fix). > > > > Thanks, > Dane > > > > From: Kevin Benton [mailto:blak...@gmail.com] > Sent: Saturday, August 23, 2014 5:47 AM > > > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required > to be run > > > > Can you disable posting of results directly from your Jenkins/Zuul setup and > have a script that just checks the log file for special markers to determine > if the vote should be FAILED/PASSED/SKIPPED? Another advantage of this > approach is that it gives you an opportunity to detect when a job just > failed to setup due to infrastructure reasons and trigger a recheck without > ever first posting a failure to gerrit. > > > > On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) > wrote: > > Thanks Edgar for updating the APIC status!!! > > Edgar and Kyle: *PLEASE NOTE** I need your understanding and > advice on the following: > > We are still stuck with a problem stemming from a design limitation of > Jenkins that prevents us from being compliant with Neutron 3rd Party CI > requirements for our DFA CI. > > The issue is that Jenkins only allows our scripts to (programmatically) > return either Success or Fail. There is no option to return "Aborted", "Not > Tested", or "Skipped". > > Why does this matter? The DFA plugin is just being introduced, and initial > DFA-enabling change sets have not yet been merged. Therefore, all other > change sets will fail our Tempest tests, since they are not DFA-enabled. > > Similarly, we were recently blocked in our APIC CI with a critical bug, > causing all change sets without this fix to fail on our APIC testbed. > > In these cases, we would like to enter a "throttled" or "partially blocked" > mode, where we would skip testing on change sets we know will fail, and (in > an ideal world) signal this shortcoming to Gerrit e.g. by returning a > "Skipped" status. Unfortunately, this option is not available in Jenkins > scripts, as Jenkins is currently designed. The only options we have > available is "Success" or all "Fail", which are both misleading. We would > also incorrectly report success or fail on one of the following test > commits: > https://review.openstack.org/#/c/114393/ > https://review.openstack.org/#/c/40296/ > > I've brought this issue up on the openstack-infra IRC, and jeblair confirmed > the Jenkins limitation, but asked me to get consensus from the Neutron > community as to this being a problem/requirement. I've also sent out an > e-mail on the Neutron ML trying to start a discussion on this problem (no > traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, > assuming there is time permitted in the open discussion. > > I'm also investigating > > For the short term, I would like to propose the following: > * We bring this up on the 3rd Party CI IRC on Monday to get a solution or > workaround, if available. If a solution is available, let's consider > including that as a hint when we come up with CI requirements for handling > CIs bocked by some critical fix. > * I'm also looking into using a REST API to cancel a Jenkins job > programmatically. > * If no solution or workaround is available, we work with infra team or with > Jenkins team to create a solution. > * Until a solution is available, for plugins which are blocked by a critical > bug, we post a status/notes indicating the plugin's situation on our 3rd > party CI status wiki, e.g.: > > Vendor Plugin/Driver Name Contact Name > Status Notes > My Vendor Name My Plugin CIMy Contact Person T > Throttled / Partially blocked / Awaiting Intial Commits > > The status/notes should be clear and understood by the Neutron team. The > console logs for change s
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Kevin: Thanks, this is a great idea! I may try just a slight variation of this concept. Maybe your idea could be the recommended way to create a 3rd party CI for plugins that are just being introduced and need to limit the scope of testing to a small set of plugin-related commits (or plugins blocked on a certain fix). Thanks, Dane From: Kevin Benton [mailto:blak...@gmail.com] Sent: Saturday, August 23, 2014 5:47 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Can you disable posting of results directly from your Jenkins/Zuul setup and have a script that just checks the log file for special markers to determine if the vote should be FAILED/PASSED/SKIPPED? Another advantage of this approach is that it gives you an opportunity to detect when a job just failed to setup due to infrastructure reasons and trigger a recheck without ever first posting a failure to gerrit. On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) mailto:lebla...@cisco.com>> wrote: Thanks Edgar for updating the APIC status!!! Edgar and Kyle: *PLEASE NOTE** I need your understanding and advice on the following: We are still stuck with a problem stemming from a design limitation of Jenkins that prevents us from being compliant with Neutron 3rd Party CI requirements for our DFA CI. The issue is that Jenkins only allows our scripts to (programmatically) return either Success or Fail. There is no option to return "Aborted", "Not Tested", or "Skipped". Why does this matter? The DFA plugin is just being introduced, and initial DFA-enabling change sets have not yet been merged. Therefore, all other change sets will fail our Tempest tests, since they are not DFA-enabled. Similarly, we were recently blocked in our APIC CI with a critical bug, causing all change sets without this fix to fail on our APIC testbed. In these cases, we would like to enter a "throttled" or "partially blocked" mode, where we would skip testing on change sets we know will fail, and (in an ideal world) signal this shortcoming to Gerrit e.g. by returning a "Skipped" status. Unfortunately, this option is not available in Jenkins scripts, as Jenkins is currently designed. The only options we have available is "Success" or all "Fail", which are both misleading. We would also incorrectly report success or fail on one of the following test commits: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I've brought this issue up on the openstack-infra IRC, and jeblair confirmed the Jenkins limitation, but asked me to get consensus from the Neutron community as to this being a problem/requirement. I've also sent out an e-mail on the Neutron ML trying to start a discussion on this problem (no traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, assuming there is time permitted in the open discussion. I'm also investigating For the short term, I would like to propose the following: * We bring this up on the 3rd Party CI IRC on Monday to get a solution or workaround, if available. If a solution is available, let's consider including that as a hint when we come up with CI requirements for handling CIs bocked by some critical fix. * I'm also looking into using a REST API to cancel a Jenkins job programmatically. * If no solution or workaround is available, we work with infra team or with Jenkins team to create a solution. * Until a solution is available, for plugins which are blocked by a critical bug, we post a status/notes indicating the plugin's situation on our 3rd party CI status wiki, e.g.: Vendor Plugin/Driver Name Contact NameStatus Notes My Vendor Name My Plugin CIMy Contact Person T Throttled / Partially blocked / Awaiting Intial Commits The status/notes should be clear and understood by the Neutron team. The console logs for change sets where the tests were skipped should also contain a message that all testing is being skipped for that commit. Note that when the DFA initial commits are merged, then this issue would go away for the DFA CI. However, this problem will reappear every time a blocking critical bug shows up for a 3rd party CI setup, or a new plugin is introduced and the hardware-enabling commits are not yet merged. (That is, until we have a solution for the Jenkins limitation). Let me know what you think. Thanks, Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com<mailto:edgar.mag...@workday.com>] Sent: Friday, August 22, 2014 1:57 PM To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Can you disable posting of results directly from your Jenkins/Zuul setup and have a script that just checks the log file for special markers to determine if the vote should be FAILED/PASSED/SKIPPED? Another advantage of this approach is that it gives you an opportunity to detect when a job just failed to setup due to infrastructure reasons and trigger a recheck without ever first posting a failure to gerrit. On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) wrote: > Thanks Edgar for updating the APIC status!!! > > Edgar and Kyle: *PLEASE NOTE** I need your understanding and > advice on the following: > > We are still stuck with a problem stemming from a design limitation of > Jenkins that prevents us from being compliant with Neutron 3rd Party CI > requirements for our DFA CI. > > The issue is that Jenkins only allows our scripts to (programmatically) > return either Success or Fail. There is no option to return "Aborted", "Not > Tested", or "Skipped". > > Why does this matter? The DFA plugin is just being introduced, and initial > DFA-enabling change sets have not yet been merged. Therefore, all other > change sets will fail our Tempest tests, since they are not DFA-enabled. > > Similarly, we were recently blocked in our APIC CI with a critical bug, > causing all change sets without this fix to fail on our APIC testbed. > > In these cases, we would like to enter a "throttled" or "partially > blocked" mode, where we would skip testing on change sets we know will > fail, and (in an ideal world) signal this shortcoming to Gerrit e.g. by > returning a "Skipped" status. Unfortunately, this option is not available > in Jenkins scripts, as Jenkins is currently designed. The only options we > have available is "Success" or all "Fail", which are both misleading. We > would also incorrectly report success or fail on one of the following test > commits: > https://review.openstack.org/#/c/114393/ > https://review.openstack.org/#/c/40296/ > > I've brought this issue up on the openstack-infra IRC, and jeblair > confirmed the Jenkins limitation, but asked me to get consensus from the > Neutron community as to this being a problem/requirement. I've also sent > out an e-mail on the Neutron ML trying to start a discussion on this > problem (no traction). I plan on bringing this up in the 3rd Party CI IRC > on Monday, assuming there is time permitted in the open discussion. > > I'm also investigating > > For the short term, I would like to propose the following: > * We bring this up on the 3rd Party CI IRC on Monday to get a solution or > workaround, if available. If a solution is available, let's consider > including that as a hint when we come up with CI requirements for handling > CIs bocked by some critical fix. > * I'm also looking into using a REST API to cancel a Jenkins job > programmatically. > * If no solution or workaround is available, we work with infra team or > with Jenkins team to create a solution. > * Until a solution is available, for plugins which are blocked by a > critical bug, we post a status/notes indicating the plugin's situation on > our 3rd party CI status wiki, e.g.: > > Vendor Plugin/Driver Name Contact Name > Status Notes > My Vendor Name My Plugin CIMy Contact Person T > Throttled / Partially blocked / Awaiting Intial Commits > > The status/notes should be clear and understood by the Neutron team. The > console logs for change sets where the tests were skipped should also > contain a message that all testing is being skipped for that commit. > > Note that when the DFA initial commits are merged, then this issue would > go away for the DFA CI. However, this problem will reappear every time a > blocking critical bug shows up for a 3rd party CI setup, or a new plugin is > introduced and the hardware-enabling commits are not yet merged. (That is, > until we have a solution for the Jenkins limitation). > > Let me know what you think. > > Thanks, > Dane > > -Original Message- > From: Edgar Magana [mailto:edgar.mag...@workday.com] > Sent: Friday, August 22, 2014 1:57 PM > To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for > usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > Sorry my bad but I just changed. > > Edgar > > On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" wrote: > > >Edgar: > > > >I'm still seeing the comment "Results are not accurate. Needs > >clarification..." > > > >Dane > > > >
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Thanks Edgar for updating the APIC status!!! Edgar and Kyle: *PLEASE NOTE** I need your understanding and advice on the following: We are still stuck with a problem stemming from a design limitation of Jenkins that prevents us from being compliant with Neutron 3rd Party CI requirements for our DFA CI. The issue is that Jenkins only allows our scripts to (programmatically) return either Success or Fail. There is no option to return "Aborted", "Not Tested", or "Skipped". Why does this matter? The DFA plugin is just being introduced, and initial DFA-enabling change sets have not yet been merged. Therefore, all other change sets will fail our Tempest tests, since they are not DFA-enabled. Similarly, we were recently blocked in our APIC CI with a critical bug, causing all change sets without this fix to fail on our APIC testbed. In these cases, we would like to enter a "throttled" or "partially blocked" mode, where we would skip testing on change sets we know will fail, and (in an ideal world) signal this shortcoming to Gerrit e.g. by returning a "Skipped" status. Unfortunately, this option is not available in Jenkins scripts, as Jenkins is currently designed. The only options we have available is "Success" or all "Fail", which are both misleading. We would also incorrectly report success or fail on one of the following test commits: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I've brought this issue up on the openstack-infra IRC, and jeblair confirmed the Jenkins limitation, but asked me to get consensus from the Neutron community as to this being a problem/requirement. I've also sent out an e-mail on the Neutron ML trying to start a discussion on this problem (no traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, assuming there is time permitted in the open discussion. I'm also investigating For the short term, I would like to propose the following: * We bring this up on the 3rd Party CI IRC on Monday to get a solution or workaround, if available. If a solution is available, let's consider including that as a hint when we come up with CI requirements for handling CIs bocked by some critical fix. * I'm also looking into using a REST API to cancel a Jenkins job programmatically. * If no solution or workaround is available, we work with infra team or with Jenkins team to create a solution. * Until a solution is available, for plugins which are blocked by a critical bug, we post a status/notes indicating the plugin's situation on our 3rd party CI status wiki, e.g.: Vendor Plugin/Driver Name Contact NameStatus Notes My Vendor Name My Plugin CIMy Contact Person T Throttled / Partially blocked / Awaiting Intial Commits The status/notes should be clear and understood by the Neutron team. The console logs for change sets where the tests were skipped should also contain a message that all testing is being skipped for that commit. Note that when the DFA initial commits are merged, then this issue would go away for the DFA CI. However, this problem will reappear every time a blocking critical bug shows up for a 3rd party CI setup, or a new plugin is introduced and the hardware-enabling commits are not yet merged. (That is, until we have a solution for the Jenkins limitation). Let me know what you think. Thanks, Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Friday, August 22, 2014 1:57 PM To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Sorry my bad but I just changed. Edgar On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" wrote: >Edgar: > >I'm still seeing the comment "Results are not accurate. Needs >clarification..." > >Dane > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Thursday, August 21, 2014 2:58 PM >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Dane, > >Wiki has been updated. > >Thanks, > >Edgar > >On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" wrote: > >>Edgar: >> >>The status on the wiki page says "Results are not accurate. Needs >>clarification from Cisco". >>Can you please tell me what we are missing? >> >>-Dane >> >>-Original Message- >>From: Dane Leblanc (leblancd) >>Sent: Tuesday, August 19, 2014 3:05 PM >>To: 'Edgar Magana'; OpenStack Developme
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Sorry my bad but I just changed. Edgar On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" wrote: >Edgar: > >I'm still seeing the comment "Results are not accurate. Needs >clarification..." > >Dane > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Thursday, August 21, 2014 2:58 PM >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for >usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Dane, > >Wiki has been updated. > >Thanks, > >Edgar > >On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" wrote: > >>Edgar: >> >>The status on the wiki page says "Results are not accurate. Needs >>clarification from Cisco". >>Can you please tell me what we are missing? >> >>-Dane >> >>-Original Message----- >>From: Dane Leblanc (leblancd) >>Sent: Tuesday, August 19, 2014 3:05 PM >>To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage >>questions) >>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>The APIC CI did run tests against that commit (after some queue latency): >> >>http://128.107.233.28:8080/job/apic/1860/ >>http://cisco-neutron-ci.cisco.com/logs/apic/1860/ >> >>But the review comments never showed up on Gerrit. This seems to be an >>intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from this >>Jenkins/Gerrit server. Whenever we disable another one of our other >>Jenkins jobs (in this case, we disabled DFA for some rework), the >>review comments sometimes stop showing up on Gerrit. >> >>-----Original Message- >>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>Sent: Tuesday, August 19, 2014 1:33 PM >>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>I was looking to one of the most recent Neutron commits: >>https://review.openstack.org/#/c/115175/ >> >> >>I could not find the APIC report. >> >>Edgar >> >>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" >>wrote: >> >>>From which commit is it missing? >>>https://review.openstack.org/#/c/114629/ >>>https://review.openstack.org/#/c/114393/ >>> >>>-Original Message- >>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>Sent: Tuesday, August 19, 2014 12:28 PM >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>>for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>Dane, >>> >>>Are you sure about it? >>>I just went to this commit and I could not find the APIC tests. >>> >>>Thanks, >>> >>>Edgar >>> >>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" >>>wrote: >>> >>>>Edgar: >>>> >>>>The Cisco APIC should be reporting results for both APIC-related and >>>>non-APIC related changes now. >>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >>>> >>>>Will you be updating the wiki page? >>>> >>>>-Dane >>>> >>>>-Original Message- >>>>From: Dane Leblanc (leblancd) >>>>Sent: Friday, August 15, 2014 8:18 PM >>>>To: OpenStack Development Mailing List (not for usage questions) >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>>required to be run >>>> >>>>Also, you can add me as a contact person for the Cisco VPNaaS driver. >>>> >>>>-----Original Message- >>>>From: Dane Leblanc (leblancd) >>>>Sent: Friday, August 15, 2014 8:14 PM >>>>To: OpenStack Development Mailing List (not for usage questions) >>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>>required to be run >>>> >>>>Edgar: >>>> >>>>For the Notes for the Cisco APIC, can you change the comment "results >>>>are fake" to something like "results are only valid for APIC-related >>>>commits"? I think this more accurately represents our current results >>>>(for reasons we chatted about on another thread). >
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Excellent Job! Thanks a lot, I have updated the wiki. Edgar From: Hemanth Ravi mailto:hemanthrav...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Thursday, August 21, 2014 at 1:37 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar, CI name: One Convergence CI The logs issue has been fixed and earlier tests were failing due to https://review.openstack.org/#/c/114146 For reference: 1. Please take a look at the vote on patchset 7 in https://review.openstack.org/#/c/114968/ 2. Logs at https://www.dropbox.com/sh/czydzz5bn2rc2lp/AABZByV8UQUIqWaZSSrZvzvDa Thanks, -hemanth On Thu, Aug 21, 2014 at 12:41 PM, Kevin Benton mailto:blak...@gmail.com>> wrote: Our system would get backed up with patches and sometimes take up to 10 hours to respond with results for a change. We should establish some maximum acceptable time to get the results for a patch. On Thu, Aug 21, 2014 at 11:59 AM, Edgar Magana mailto:edgar.mag...@workday.com>> wrote: Maximum time to vote, can you clarify? Edgar From: Kevin Benton mailto:blak...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Tuesday, August 19, 2014 at 1:11 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run I also added a two more nodes to our cluster to reduce the delay. Can we establish a maximum time to vote on the wiki? On Aug 19, 2014 9:39 AM, "Edgar Magana" mailto:edgar.mag...@workday.com>> wrote: Kevin, I just verified, Thanks a lot. Edgar From: Kevin Benton mailto:blak...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Friday, August 15, 2014 at 4:56 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run You didn't wait long enough for the Big Switch CI to reach your negative test. :-) Currently the CI system is backed by about 100 patches to push responses out ~22 hours. This does bring up an old discussion that was never resolved. What is the minimum expected response time for CI systems? On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana mailto:edgar.mag...@workday.com>> wrote: Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" mailto:mest...@mestery.com>> wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below >and make sure you are running appropriate tests. If you have >questions, the weekly third-party meeting [2] is a great place to ask >questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/op
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Edgar: I'm still seeing the comment "Results are not accurate. Needs clarification..." Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Thursday, August 21, 2014 2:58 PM To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Dane, Wiki has been updated. Thanks, Edgar On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" wrote: >Edgar: > >The status on the wiki page says "Results are not accurate. Needs >clarification from Cisco". >Can you please tell me what we are missing? > >-Dane > >-Original Message- >From: Dane Leblanc (leblancd) >Sent: Tuesday, August 19, 2014 3:05 PM >To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage >questions) >Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >The APIC CI did run tests against that commit (after some queue latency): > >http://128.107.233.28:8080/job/apic/1860/ >http://cisco-neutron-ci.cisco.com/logs/apic/1860/ > >But the review comments never showed up on Gerrit. This seems to be an >intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from this >Jenkins/Gerrit server. Whenever we disable another one of our other >Jenkins jobs (in this case, we disabled DFA for some rework), the >review comments sometimes stop showing up on Gerrit. > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Tuesday, August 19, 2014 1:33 PM >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >I was looking to one of the most recent Neutron commits: >https://review.openstack.org/#/c/115175/ > > >I could not find the APIC report. > >Edgar > >On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" wrote: > >>From which commit is it missing? >>https://review.openstack.org/#/c/114629/ >>https://review.openstack.org/#/c/114393/ >> >>-Original Message----- >>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>Sent: Tuesday, August 19, 2014 12:28 PM >>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Dane, >> >>Are you sure about it? >>I just went to this commit and I could not find the APIC tests. >> >>Thanks, >> >>Edgar >> >>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" >>wrote: >> >>>Edgar: >>> >>>The Cisco APIC should be reporting results for both APIC-related and >>>non-APIC related changes now. >>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >>> >>>Will you be updating the wiki page? >>> >>>-Dane >>> >>>-Original Message- >>>From: Dane Leblanc (leblancd) >>>Sent: Friday, August 15, 2014 8:18 PM >>>To: OpenStack Development Mailing List (not for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>Also, you can add me as a contact person for the Cisco VPNaaS driver. >>> >>>-Original Message- >>>From: Dane Leblanc (leblancd) >>>Sent: Friday, August 15, 2014 8:14 PM >>>To: OpenStack Development Mailing List (not for usage questions) >>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>Edgar: >>> >>>For the Notes for the Cisco APIC, can you change the comment "results >>>are fake" to something like "results are only valid for APIC-related >>>commits"? I think this more accurately represents our current results >>>(for reasons we chatted about on another thread). >>> >>>Thanks, >>>Dane >>> >>>-Original Message- >>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>Sent: Friday, August 15, 2014 6:36 PM >>>To: OpenStack Development Mailing List (not for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>>Importance: High >>> >>>Team, >>> >>>I did a quick audit on the Neutron CI. Very sad results. Only few >>>plugins and drivers are
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Edgar, CI name: One Convergence CI The logs issue has been fixed and earlier tests were failing due to https://review.openstack.org/#/c/114146 For reference: 1. Please take a look at the vote on patchset 7 in https://review.openstack.org/#/c/114968/ 2. Logs at https://www.dropbox.com/sh/czydzz5bn2rc2lp/AABZByV8UQUIqWaZSSrZvzvDa Thanks, -hemanth On Thu, Aug 21, 2014 at 12:41 PM, Kevin Benton wrote: > Our system would get backed up with patches and sometimes take up to 10 > hours to respond with results for a change. > We should establish some maximum acceptable time to get the results for a > patch. > > > On Thu, Aug 21, 2014 at 11:59 AM, Edgar Magana > wrote: > >> Maximum time to vote, can you clarify? >> >> Edgar >> >> From: Kevin Benton >> Reply-To: "OpenStack Development Mailing List (not for usage questions)" >> >> Date: Tuesday, August 19, 2014 at 1:11 PM >> >> To: "OpenStack Development Mailing List (not for usage questions)" < >> openstack-dev@lists.openstack.org> >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >> required to be run >> >> I also added a two more nodes to our cluster to reduce the delay. Can >> we establish a maximum time to vote on the wiki? >> On Aug 19, 2014 9:39 AM, "Edgar Magana" wrote: >> >>> Kevin, >>> >>> I just verified, Thanks a lot. >>> >>> Edgar >>> >>> From: Kevin Benton >>> Reply-To: "OpenStack Development Mailing List (not for usage >>> questions)" >>> Date: Friday, August 15, 2014 at 4:56 PM >>> To: "OpenStack Development Mailing List (not for usage questions)" < >>> openstack-dev@lists.openstack.org> >>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>> required to be run >>> >>> You didn't wait long enough for the Big Switch CI to reach your >>> negative test. :-) >>> Currently the CI system is backed by about 100 patches to push >>> responses out ~22 hours. >>> >>> This does bring up an old discussion that was never resolved. >>> What is the minimum expected response time for CI systems? >>> >>> >>> On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana >>> wrote: >>> >>>> Team, >>>> >>>> I did a quick audit on the Neutron CI. Very sad results. Only few >>>> plugins >>>> and drivers are running properly and testing all Neutron commits. >>>> I created a report here: >>>> >>>> https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin >>>> _and_Drivers >>>> >>>> >>>> We will discuss the actions to take on the next Neutron IRC meeting. So >>>> please, reach me out to clarify what is the status of your CI. >>>> I had two commits to quickly verify the CI reliability: >>>> >>>> https://review.openstack.org/#/c/114393/ >>>> >>>> https://review.openstack.org/#/c/40296/ >>>> >>>> >>>> I would expect all plugins and drivers passing on the first one and >>>> failing for the second but I got so many surprises. >>>> >>>> Neutron code quality and reliability is a top priority, if you ignore >>>> this >>>> report that plugin/driver will be candidate to be remove from Neutron >>>> tree. >>>> >>>> Cheers, >>>> >>>> Edgar >>>> >>>> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>>> job! >>>> >>>> >>>> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >>>> >>>> >Folks, I'm not sure if all CI accounts are running sufficient tests. >>>> >Per the requirements wiki page here [1], everyone needs to be running >>>> >more than just Tempest API tests, which I still see most neutron >>>> >third-party CI setups doing. I'd like to ask everyone who operates a >>>> >third-party CI account for Neutron to please look at the link below >>>> >and make sure you are running appropriate tests. If you have >>>> >questions, the weekly third-party meeting [2] is a great place to ask >>>> >questions. >>>> > >>>> >Thanks, >>>> >Kyle >>>> > >>>> >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Our system would get backed up with patches and sometimes take up to 10 hours to respond with results for a change. We should establish some maximum acceptable time to get the results for a patch. On Thu, Aug 21, 2014 at 11:59 AM, Edgar Magana wrote: > Maximum time to vote, can you clarify? > > Edgar > > From: Kevin Benton > Reply-To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Date: Tuesday, August 19, 2014 at 1:11 PM > > To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > I also added a two more nodes to our cluster to reduce the delay. Can > we establish a maximum time to vote on the wiki? > On Aug 19, 2014 9:39 AM, "Edgar Magana" wrote: > >> Kevin, >> >> I just verified, Thanks a lot. >> >> Edgar >> >> From: Kevin Benton >> Reply-To: "OpenStack Development Mailing List (not for usage questions)" >> >> Date: Friday, August 15, 2014 at 4:56 PM >> To: "OpenStack Development Mailing List (not for usage questions)" < >> openstack-dev@lists.openstack.org> >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >> required to be run >> >> You didn't wait long enough for the Big Switch CI to reach your >> negative test. :-) >> Currently the CI system is backed by about 100 patches to push responses >> out ~22 hours. >> >> This does bring up an old discussion that was never resolved. >> What is the minimum expected response time for CI systems? >> >> >> On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana >> wrote: >> >>> Team, >>> >>> I did a quick audit on the Neutron CI. Very sad results. Only few plugins >>> and drivers are running properly and testing all Neutron commits. >>> I created a report here: >>> >>> https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin >>> _and_Drivers >>> >>> >>> We will discuss the actions to take on the next Neutron IRC meeting. So >>> please, reach me out to clarify what is the status of your CI. >>> I had two commits to quickly verify the CI reliability: >>> >>> https://review.openstack.org/#/c/114393/ >>> >>> https://review.openstack.org/#/c/40296/ >>> >>> >>> I would expect all plugins and drivers passing on the first one and >>> failing for the second but I got so many surprises. >>> >>> Neutron code quality and reliability is a top priority, if you ignore >>> this >>> report that plugin/driver will be candidate to be remove from Neutron >>> tree. >>> >>> Cheers, >>> >>> Edgar >>> >>> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>> job! >>> >>> >>> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >>> >>> >Folks, I'm not sure if all CI accounts are running sufficient tests. >>> >Per the requirements wiki page here [1], everyone needs to be running >>> >more than just Tempest API tests, which I still see most neutron >>> >third-party CI setups doing. I'd like to ask everyone who operates a >>> >third-party CI account for Neutron to please look at the link below >>> >and make sure you are running appropriate tests. If you have >>> >questions, the weekly third-party meeting [2] is a great place to ask >>> >questions. >>> > >>> >Thanks, >>> >Kyle >>> > >>> >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>> >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>> > >>> >___ >>> >OpenStack-dev mailing list >>> >OpenStack-dev@lists.openstack.org >>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> >> -- >> Kevin Benton >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Maximum time to vote, can you clarify? Edgar From: Kevin Benton mailto:blak...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Tuesday, August 19, 2014 at 1:11 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run I also added a two more nodes to our cluster to reduce the delay. Can we establish a maximum time to vote on the wiki? On Aug 19, 2014 9:39 AM, "Edgar Magana" mailto:edgar.mag...@workday.com>> wrote: Kevin, I just verified, Thanks a lot. Edgar From: Kevin Benton mailto:blak...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Friday, August 15, 2014 at 4:56 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run You didn't wait long enough for the Big Switch CI to reach your negative test. :-) Currently the CI system is backed by about 100 patches to push responses out ~22 hours. This does bring up an old discussion that was never resolved. What is the minimum expected response time for CI systems? On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana mailto:edgar.mag...@workday.com>> wrote: Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" mailto:mest...@mestery.com>> wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below >and make sure you are running appropriate tests. If you have >questions, the weekly third-party meeting [2] is a great place to ask >questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Dane, Wiki has been updated. Thanks, Edgar On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" wrote: >Edgar: > >The status on the wiki page says "Results are not accurate. Needs >clarification from Cisco". >Can you please tell me what we are missing? > >-Dane > >-Original Message- >From: Dane Leblanc (leblancd) >Sent: Tuesday, August 19, 2014 3:05 PM >To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage >questions) >Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >The APIC CI did run tests against that commit (after some queue latency): > >http://128.107.233.28:8080/job/apic/1860/ >http://cisco-neutron-ci.cisco.com/logs/apic/1860/ > >But the review comments never showed up on Gerrit. This seems to be an >intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from this >Jenkins/Gerrit server. Whenever we disable another one of our other >Jenkins jobs (in this case, we disabled DFA for some rework), the review >comments sometimes stop showing up on Gerrit. > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Tuesday, August 19, 2014 1:33 PM >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for >usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >I was looking to one of the most recent Neutron commits: >https://review.openstack.org/#/c/115175/ > > >I could not find the APIC report. > >Edgar > >On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" wrote: > >>From which commit is it missing? >>https://review.openstack.org/#/c/114629/ >>https://review.openstack.org/#/c/114393/ >> >>-Original Message----- >>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>Sent: Tuesday, August 19, 2014 12:28 PM >>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >>for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Dane, >> >>Are you sure about it? >>I just went to this commit and I could not find the APIC tests. >> >>Thanks, >> >>Edgar >> >>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" >>wrote: >> >>>Edgar: >>> >>>The Cisco APIC should be reporting results for both APIC-related and >>>non-APIC related changes now. >>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >>> >>>Will you be updating the wiki page? >>> >>>-Dane >>> >>>-Original Message- >>>From: Dane Leblanc (leblancd) >>>Sent: Friday, August 15, 2014 8:18 PM >>>To: OpenStack Development Mailing List (not for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>Also, you can add me as a contact person for the Cisco VPNaaS driver. >>> >>>-Original Message- >>>From: Dane Leblanc (leblancd) >>>Sent: Friday, August 15, 2014 8:14 PM >>>To: OpenStack Development Mailing List (not for usage questions) >>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>> >>>Edgar: >>> >>>For the Notes for the Cisco APIC, can you change the comment "results >>>are fake" to something like "results are only valid for APIC-related >>>commits"? I think this more accurately represents our current results >>>(for reasons we chatted about on another thread). >>> >>>Thanks, >>>Dane >>> >>>-Original Message- >>>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>>Sent: Friday, August 15, 2014 6:36 PM >>>To: OpenStack Development Mailing List (not for usage questions) >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>>required to be run >>>Importance: High >>> >>>Team, >>> >>>I did a quick audit on the Neutron CI. Very sad results. Only few >>>plugins and drivers are running properly and testing all Neutron >>>commits. >>>I created a report here: >>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_P >>>l >>>ugi >>>n >>>_and_Drivers >>> >>> >>>We will discuss the actions to take on the next Neutron IRC meeting. >>>So please, reach me out to clarify what is the status of your CI. >&
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Edgar: The status on the wiki page says "Results are not accurate. Needs clarification from Cisco". Can you please tell me what we are missing? -Dane -Original Message- From: Dane Leblanc (leblancd) Sent: Tuesday, August 19, 2014 3:05 PM To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage questions) Subject: RE: [openstack-dev] [neutron] [third-party] What tests are required to be run The APIC CI did run tests against that commit (after some queue latency): http://128.107.233.28:8080/job/apic/1860/ http://cisco-neutron-ci.cisco.com/logs/apic/1860/ But the review comments never showed up on Gerrit. This seems to be an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from this Jenkins/Gerrit server. Whenever we disable another one of our other Jenkins jobs (in this case, we disabled DFA for some rework), the review comments sometimes stop showing up on Gerrit. -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Tuesday, August 19, 2014 1:33 PM To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run I was looking to one of the most recent Neutron commits: https://review.openstack.org/#/c/115175/ I could not find the APIC report. Edgar On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" wrote: >From which commit is it missing? >https://review.openstack.org/#/c/114629/ >https://review.openstack.org/#/c/114393/ > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Tuesday, August 19, 2014 12:28 PM >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Dane, > >Are you sure about it? >I just went to this commit and I could not find the APIC tests. > >Thanks, > >Edgar > >On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" wrote: > >>Edgar: >> >>The Cisco APIC should be reporting results for both APIC-related and >>non-APIC related changes now. >>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >> >>Will you be updating the wiki page? >> >>-Dane >> >>-----Original Message----- >>From: Dane Leblanc (leblancd) >>Sent: Friday, August 15, 2014 8:18 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Also, you can add me as a contact person for the Cisco VPNaaS driver. >> >>-----Original Message----- >>From: Dane Leblanc (leblancd) >>Sent: Friday, August 15, 2014 8:14 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Edgar: >> >>For the Notes for the Cisco APIC, can you change the comment "results >>are fake" to something like "results are only valid for APIC-related >>commits"? I think this more accurately represents our current results >>(for reasons we chatted about on another thread). >> >>Thanks, >>Dane >> >>-Original Message- >>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>Sent: Friday, August 15, 2014 6:36 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >>Importance: High >> >>Team, >> >>I did a quick audit on the Neutron CI. Very sad results. Only few >>plugins and drivers are running properly and testing all Neutron commits. >>I created a report here: >>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_P >>l >>ugi >>n >>_and_Drivers >> >> >>We will discuss the actions to take on the next Neutron IRC meeting. >>So please, reach me out to clarify what is the status of your CI. >>I had two commits to quickly verify the CI reliability: >> >>https://review.openstack.org/#/c/114393/ >> >>https://review.openstack.org/#/c/40296/ >> >> >>I would expect all plugins and drivers passing on the first one and >>failing for the second but I got so many surprises. >> >>Neutron code quality and reliability is a top priority, if you ignore >>this report that plugin/driver will be candidate to be remove from >>Neutron tree. >> >>Cheers, >> >>Edgar >> >>P.s. I hate to be the inquisitor her
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
I also added a two more nodes to our cluster to reduce the delay. Can we establish a maximum time to vote on the wiki? On Aug 19, 2014 9:39 AM, "Edgar Magana" wrote: > Kevin, > > I just verified, Thanks a lot. > > Edgar > > From: Kevin Benton > Reply-To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Date: Friday, August 15, 2014 at 4:56 PM > To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > You didn't wait long enough for the Big Switch CI to reach your > negative test. :-) > Currently the CI system is backed by about 100 patches to push responses > out ~22 hours. > > This does bring up an old discussion that was never resolved. > What is the minimum expected response time for CI systems? > > > On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana > wrote: > >> Team, >> >> I did a quick audit on the Neutron CI. Very sad results. Only few plugins >> and drivers are running properly and testing all Neutron commits. >> I created a report here: >> >> https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin >> _and_Drivers >> >> >> We will discuss the actions to take on the next Neutron IRC meeting. So >> please, reach me out to clarify what is the status of your CI. >> I had two commits to quickly verify the CI reliability: >> >> https://review.openstack.org/#/c/114393/ >> >> https://review.openstack.org/#/c/40296/ >> >> >> I would expect all plugins and drivers passing on the first one and >> failing for the second but I got so many surprises. >> >> Neutron code quality and reliability is a top priority, if you ignore this >> report that plugin/driver will be candidate to be remove from Neutron >> tree. >> >> Cheers, >> >> Edgar >> >> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >> job! >> >> >> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >> >> >Folks, I'm not sure if all CI accounts are running sufficient tests. >> >Per the requirements wiki page here [1], everyone needs to be running >> >more than just Tempest API tests, which I still see most neutron >> >third-party CI setups doing. I'd like to ask everyone who operates a >> >third-party CI account for Neutron to please look at the link below >> >and make sure you are running appropriate tests. If you have >> >questions, the weekly third-party meeting [2] is a great place to ask >> >questions. >> > >> >Thanks, >> >Kyle >> > >> >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >> >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> > >> >___ >> >OpenStack-dev mailing list >> >OpenStack-dev@lists.openstack.org >> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > > -- > Kevin Benton > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
The APIC CI did run tests against that commit (after some queue latency): http://128.107.233.28:8080/job/apic/1860/ http://cisco-neutron-ci.cisco.com/logs/apic/1860/ But the review comments never showed up on Gerrit. This seems to be an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from this Jenkins/Gerrit server. Whenever we disable another one of our other Jenkins jobs (in this case, we disabled DFA for some rework), the review comments sometimes stop showing up on Gerrit. -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Tuesday, August 19, 2014 1:33 PM To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run I was looking to one of the most recent Neutron commits: https://review.openstack.org/#/c/115175/ I could not find the APIC report. Edgar On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" wrote: >From which commit is it missing? >https://review.openstack.org/#/c/114629/ >https://review.openstack.org/#/c/114393/ > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Tuesday, August 19, 2014 12:28 PM >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not >for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Dane, > >Are you sure about it? >I just went to this commit and I could not find the APIC tests. > >Thanks, > >Edgar > >On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" wrote: > >>Edgar: >> >>The Cisco APIC should be reporting results for both APIC-related and >>non-APIC related changes now. >>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >> >>Will you be updating the wiki page? >> >>-Dane >> >>-Original Message----- >>From: Dane Leblanc (leblancd) >>Sent: Friday, August 15, 2014 8:18 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Also, you can add me as a contact person for the Cisco VPNaaS driver. >> >>-Original Message----- >>From: Dane Leblanc (leblancd) >>Sent: Friday, August 15, 2014 8:14 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Edgar: >> >>For the Notes for the Cisco APIC, can you change the comment "results >>are fake" to something like "results are only valid for APIC-related >>commits"? I think this more accurately represents our current results >>(for reasons we chatted about on another thread). >> >>Thanks, >>Dane >> >>-Original Message- >>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>Sent: Friday, August 15, 2014 6:36 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >>Importance: High >> >>Team, >> >>I did a quick audit on the Neutron CI. Very sad results. Only few >>plugins and drivers are running properly and testing all Neutron commits. >>I created a report here: >>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_P >>l >>ugi >>n >>_and_Drivers >> >> >>We will discuss the actions to take on the next Neutron IRC meeting. >>So please, reach me out to clarify what is the status of your CI. >>I had two commits to quickly verify the CI reliability: >> >>https://review.openstack.org/#/c/114393/ >> >>https://review.openstack.org/#/c/40296/ >> >> >>I would expect all plugins and drivers passing on the first one and >>failing for the second but I got so many surprises. >> >>Neutron code quality and reliability is a top priority, if you ignore >>this report that plugin/driver will be candidate to be remove from >>Neutron tree. >> >>Cheers, >> >>Edgar >> >>P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>job! >> >> >>On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >> >>>Folks, I'm not sure if all CI accounts are running sufficient tests. >>>Per the requirements wiki page here [1], everyone needs to be running >>>more than just Tempest API tests, which I still see most neut
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
I was looking to one of the most recent Neutron commits: https://review.openstack.org/#/c/115175/ I could not find the APIC report. Edgar On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" wrote: >From which commit is it missing? >https://review.openstack.org/#/c/114629/ >https://review.openstack.org/#/c/114393/ > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Tuesday, August 19, 2014 12:28 PM >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for >usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Dane, > >Are you sure about it? >I just went to this commit and I could not find the APIC tests. > >Thanks, > >Edgar > >On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" wrote: > >>Edgar: >> >>The Cisco APIC should be reporting results for both APIC-related and >>non-APIC related changes now. >>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). >> >>Will you be updating the wiki page? >> >>-Dane >> >>-Original Message----- >>From: Dane Leblanc (leblancd) >>Sent: Friday, August 15, 2014 8:18 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Also, you can add me as a contact person for the Cisco VPNaaS driver. >> >>-Original Message----- >>From: Dane Leblanc (leblancd) >>Sent: Friday, August 15, 2014 8:14 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >> >>Edgar: >> >>For the Notes for the Cisco APIC, can you change the comment "results >>are fake" to something like "results are only valid for APIC-related >>commits"? I think this more accurately represents our current results >>(for reasons we chatted about on another thread). >> >>Thanks, >>Dane >> >>-Original Message- >>From: Edgar Magana [mailto:edgar.mag...@workday.com] >>Sent: Friday, August 15, 2014 6:36 PM >>To: OpenStack Development Mailing List (not for usage questions) >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >>required to be run >>Importance: High >> >>Team, >> >>I did a quick audit on the Neutron CI. Very sad results. Only few >>plugins and drivers are running properly and testing all Neutron commits. >>I created a report here: >>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Pl >>ugi >>n >>_and_Drivers >> >> >>We will discuss the actions to take on the next Neutron IRC meeting. So >>please, reach me out to clarify what is the status of your CI. >>I had two commits to quickly verify the CI reliability: >> >>https://review.openstack.org/#/c/114393/ >> >>https://review.openstack.org/#/c/40296/ >> >> >>I would expect all plugins and drivers passing on the first one and >>failing for the second but I got so many surprises. >> >>Neutron code quality and reliability is a top priority, if you ignore >>this report that plugin/driver will be candidate to be remove from >>Neutron tree. >> >>Cheers, >> >>Edgar >> >>P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>job! >> >> >>On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >> >>>Folks, I'm not sure if all CI accounts are running sufficient tests. >>>Per the requirements wiki page here [1], everyone needs to be running >>>more than just Tempest API tests, which I still see most neutron >>>third-party CI setups doing. I'd like to ask everyone who operates a >>>third-party CI account for Neutron to please look at the link below >>>and make sure you are running appropriate tests. If you have >>>questions, the weekly third-party meeting [2] is a great place to ask >>>questions. >>> >>>Thanks, >>>Kyle >>> >>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>> >>>___ >>>OpenStack-dev mailing list >>>OpenStack-dev@lists.openstack.org >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
>From which commit is it missing? https://review.openstack.org/#/c/114629/ https://review.openstack.org/#/c/114393/ -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Tuesday, August 19, 2014 12:28 PM To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Dane, Are you sure about it? I just went to this commit and I could not find the APIC tests. Thanks, Edgar On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" wrote: >Edgar: > >The Cisco APIC should be reporting results for both APIC-related and >non-APIC related changes now. >(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). > >Will you be updating the wiki page? > >-Dane > >-Original Message- >From: Dane Leblanc (leblancd) >Sent: Friday, August 15, 2014 8:18 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Also, you can add me as a contact person for the Cisco VPNaaS driver. > >-Original Message- >From: Dane Leblanc (leblancd) >Sent: Friday, August 15, 2014 8:14 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Edgar: > >For the Notes for the Cisco APIC, can you change the comment "results >are fake" to something like "results are only valid for APIC-related >commits"? I think this more accurately represents our current results >(for reasons we chatted about on another thread). > >Thanks, >Dane > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Friday, August 15, 2014 6:36 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run >Importance: High > >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few >plugins and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Pl >ugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore >this report that plugin/driver will be candidate to be remove from >Neutron tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Kevin, I just verified, Thanks a lot. Edgar From: Kevin Benton mailto:blak...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Friday, August 15, 2014 at 4:56 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run You didn't wait long enough for the Big Switch CI to reach your negative test. :-) Currently the CI system is backed by about 100 patches to push responses out ~22 hours. This does bring up an old discussion that was never resolved. What is the minimum expected response time for CI systems? On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana mailto:edgar.mag...@workday.com>> wrote: Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" mailto:mest...@mestery.com>> wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below >and make sure you are running appropriate tests. If you have >questions, the weekly third-party meeting [2] is a great place to ask >questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Dane, It is not my intention to damage anyone's image or work. I ran a test and I noticed that APIC CI was giving a PASSED result when it was not even running anything, for me that means "fake". I just responded to your other email, I adopt see APIC CI running and I can't change the wiki until I verify it. I am online and you can contact me on IRC. Thanks, Edgar On 8/15/14, 5:13 PM, "Dane Leblanc (leblancd)" wrote: >Edgar: > >For the Notes for the Cisco APIC, can you change the comment "results are >fake" to something like "results are only valid for APIC-related >commits"? I think this more accurately represents our current results >(for reasons we chatted about on another thread). > >Thanks, >Dane > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Friday, August 15, 2014 6:36 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run >Importance: High > >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore >this report that plugin/driver will be candidate to be remove from >Neutron tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below and >>make sure you are running appropriate tests. If you have questions, the >>weekly third-party meeting [2] is a great place to ask questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Great! I will run another test later today. Thanks, Edgar From: Ichihara Hirofumi mailto:ichihara.hirof...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Friday, August 15, 2014 at 5:33 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Hi, Edgar MetapluginCI didn't run for document changes. I changed filter so that CI run for ALL changes include documents. Thanks, Hirofumi 2014-08-16 7:35 GMT+09:00 Edgar Magana mailto:edgar.mag...@workday.com>>: Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" mailto:mest...@mestery.com>> wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below >and make sure you are running appropriate tests. If you have >questions, the weekly third-party meeting [2] is a great place to ask >questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Thanks a lot, Brocade CI is looking good. Cheers, Edgar On 8/15/14, 6:22 PM, "Karthik Natarajan" wrote: >Hi Edgar, > >Brocade Vyatta CI is reporting the results and providing log links. >For Brocade Vyatta Plugin, I have updated my name as the owner. >For Brocade VDX Plugin, Shiv Haris is the owner. >Please let me know if you have any questions. > >Thanks, >Karthik > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Friday, August 15, 2014 3:59 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run >Importance: High > >Correction > >The right links are: > >https://review.openstack.org/#/c/114629/ > >https://review.openstack.org/#/c/114393/ > > >Not this one: >https://review.openstack.org/#/c/40296/ > > >Edgar > >On 8/15/14, 3:35 PM, "Edgar Magana" wrote: > >>Team, >> >>I did a quick audit on the Neutron CI. Very sad results. Only few >>plugins and drivers are running properly and testing all Neutron commits. >>I created a report here: >>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Pl >>ugi >>n >>_and_Drivers >> >> >>We will discuss the actions to take on the next Neutron IRC meeting. So >>please, reach me out to clarify what is the status of your CI. >>I had two commits to quickly verify the CI reliability: >> >>https://review.openstack.org/#/c/114393/ >> >>https://review.openstack.org/#/c/40296/ >> >> >>I would expect all plugins and drivers passing on the first one and >>failing for the second but I got so many surprises. >> >>Neutron code quality and reliability is a top priority, if you ignore >>this report that plugin/driver will be candidate to be remove from >>Neutron tree. >> >>Cheers, >> >>Edgar >> >>P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>job! >> >> >>On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >> >>>Folks, I'm not sure if all CI accounts are running sufficient tests. >>>Per the requirements wiki page here [1], everyone needs to be running >>>more than just Tempest API tests, which I still see most neutron >>>third-party CI setups doing. I'd like to ask everyone who operates a >>>third-party CI account for Neutron to please look at the link below >>>and make sure you are running appropriate tests. If you have >>>questions, the weekly third-party meeting [2] is a great place to ask >>>questions. >>> >>>Thanks, >>>Kyle >>> >>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>> >>>___ >>>OpenStack-dev mailing list >>>OpenStack-dev@lists.openstack.org >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
I just updated the wiki, I will run a verification by end of the day. Thanks, Edgar On 8/16/14, 11:34 PM, "trinath.soman...@freescale.com" wrote: >Hi Edgar, > >Freescale CI is reporting the results for ML2 Mechanism driver (J-1) and >FWaaS Plugin (to be approved for J-3). >I'm the owner for this CI. > >The Wiki page for this CI is >https://wiki.openstack.org/wiki/ThirdPartySystems/Freescale_CI. > >-- >Trinath Somanchi - B39208 >trinath.soman...@freescale.com | extn: 4048 > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Saturday, August 16, 2014 4:06 AM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run >Importance: High > >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore >this report that plugin/driver will be candidate to be remove from >Neutron tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below and >>make sure you are running appropriate tests. If you have questions, the >>weekly third-party meeting [2] is a great place to ask questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Dane, Are you sure about it? I just went to this commit and I could not find the APIC tests. Thanks, Edgar On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" wrote: >Edgar: > >The Cisco APIC should be reporting results for both APIC-related and >non-APIC related changes now. >(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). > >Will you be updating the wiki page? > >-Dane > >-Original Message- >From: Dane Leblanc (leblancd) >Sent: Friday, August 15, 2014 8:18 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Also, you can add me as a contact person for the Cisco VPNaaS driver. > >-Original Message- >From: Dane Leblanc (leblancd) >Sent: Friday, August 15, 2014 8:14 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: RE: [openstack-dev] [neutron] [third-party] What tests are >required to be run > >Edgar: > >For the Notes for the Cisco APIC, can you change the comment "results are >fake" to something like "results are only valid for APIC-related >commits"? I think this more accurately represents our current results >(for reasons we chatted about on another thread). > >Thanks, >Dane > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Friday, August 15, 2014 6:36 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run >Importance: High > >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore >this report that plugin/driver will be candidate to be remove from >Neutron tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below and >>make sure you are running appropriate tests. If you have questions, the >>weekly third-party meeting [2] is a great place to ask questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
I just added it to the wiki page. Thanks, Edgar On 8/17/14, 8:49 PM, "balaj...@freescale.com" wrote: >Hi Edgar, > >Freescale CI is not listed in the below report: > >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Pl > >We are following all the requirements of CI setup and as well >participating in the IRC Meeting. Can you please let us know if we are >missing any other requirements of CI Setup. > >Regards, >Balaji.P > > > >> -Original Message- >> From: trinath.soman...@freescale.com >> [mailto:trinath.soman...@freescale.com] >> Sent: Sunday, August 17, 2014 12:04 PM >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >> required to be run >> >> Hi Edgar, >> >> Freescale CI is reporting the results for ML2 Mechanism driver (J-1) and >> FWaaS Plugin (to be approved for J-3). >> I'm the owner for this CI. >> >> The Wiki page for this CI is >> https://wiki.openstack.org/wiki/ThirdPartySystems/Freescale_CI. >> >> -- >> Trinath Somanchi - B39208 >> trinath.soman...@freescale.com | extn: 4048 >> >> -Original Message- >> From: Edgar Magana [mailto:edgar.mag...@workday.com] >> Sent: Saturday, August 16, 2014 4:06 AM >> To: OpenStack Development Mailing List (not for usage questions) >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >> required to be run >> Importance: High >> >> Team, >> >> I did a quick audit on the Neutron CI. Very sad results. Only few >>plugins >> and drivers are running properly and testing all Neutron commits. >> I created a report here: >> >>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plug >> in >> _and_Drivers >> >> >> We will discuss the actions to take on the next Neutron IRC meeting. So >> please, reach me out to clarify what is the status of your CI. >> I had two commits to quickly verify the CI reliability: >> >> https://review.openstack.org/#/c/114393/ >> >> https://review.openstack.org/#/c/40296/ >> >> >> I would expect all plugins and drivers passing on the first one and >> failing for the second but I got so many surprises. >> >> Neutron code quality and reliability is a top priority, if you ignore >> this report that plugin/driver will be candidate to be remove from >> Neutron tree. >> >> Cheers, >> >> Edgar >> >> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >> job! >> >> >> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >> >> >Folks, I'm not sure if all CI accounts are running sufficient tests. >> >Per the requirements wiki page here [1], everyone needs to be running >> >more than just Tempest API tests, which I still see most neutron >> >third-party CI setups doing. I'd like to ask everyone who operates a >> >third-party CI account for Neutron to please look at the link below and >> >make sure you are running appropriate tests. If you have questions, the >> >weekly third-party meeting [2] is a great place to ask questions. >> > >> >Thanks, >> >Kyle >> > >> >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >> >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> > >> >___ >> >OpenStack-dev mailing list >> >OpenStack-dev@lists.openstack.org >> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hemanth, What is your system reference name? Please, provide me a link to the latest tempest logs. Thanks, Edgar From: Hemanth Ravi mailto:hemanthrav...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Monday, August 18, 2014 at 12:24 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>>, Kedar K mailto:kedar.kulka...@oneconvergence.com>>, Deepak Gupta mailto:deepak.gu...@oneconvergence.com>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar, Our CI is running the tests (non voting), but don't see it listed on the review for any patch. Is this due to missing logs? I would like to confirm this is the issue, will resolve this. Thanks, -hemanth On Mon, Aug 18, 2014 at 8:35 AM, Edgar Magana mailto:edgar.mag...@workday.com>> wrote: Thank you Akihiro. I will propose a better organization for this section. Stay tune! Edgar On 8/17/14, 10:53 PM, "Akihiro Motoki" mailto:mot...@da.jp.nec.com>> wrote: > >On 2014/08/18 0:12, Kyle Mestery wrote: >> On Fri, Aug 15, 2014 at 5:35 PM, Edgar Magana >>mailto:edgar.mag...@workday.com>> wrote: >>> Team, >>> >>> I did a quick audit on the Neutron CI. Very sad results. Only few >>>plugins >>> and drivers are running properly and testing all Neutron commits. >>> I created a report here: >>> >>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plu >>>gin >>> _and_Drivers >>> >> Can you link this and/or move it to this page: >> >> https://wiki.openstack.org/wiki/NeutronPlugins >> >> This is under the "NeutronPolicies" wiki page which I did at the start >> of Juno. This tracks all policies and procedures for Neutron, and >> there's a Plugins page (which I linked to above) where this should >> land. > >I just added the link Neutron_Plugins_and_Drivers#Existing_Plugin >to NeutronPlugins wiki. > >The wiki pages NeutronPlugins and Neutron_Plugins_and_Drivers >cover the similar contents. According to the history of the page, >the latter one was created by Mark at Nov 2013 (beginning of Icehouse >cycle). >It seems better to merge these two pages to avoid the confusion. > >Akihiro > >> >>> >>> We will discuss the actions to take on the next Neutron IRC meeting. So >>> please, reach me out to clarify what is the status of your CI. >>> I had two commits to quickly verify the CI reliability: >>> >>> https://review.openstack.org/#/c/114393/ >>> >>> https://review.openstack.org/#/c/40296/ >>> >>> >>> I would expect all plugins and drivers passing on the first one and >>> failing for the second but I got so many surprises. >>> >>> Neutron code quality and reliability is a top priority, if you ignore >>>this >>> report that plugin/driver will be candidate to be remove from Neutron >>>tree. >>> >>> Cheers, >>> >>> Edgar >>> >>> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>>job! >>> >> Thanks for sending this out Edgar and doing this analysis! Can you >> please put an agenda item on Monday's meeting to discuss this? I won't >> be at the meeting as I'm on PTO (Mark is running the meeting in my >> absence), but I'd like the team to discuss this and allow all >> third-party people a chance to be there and share their feelings here. >> >> Thanks, >> Kyle >> >>> >>> On 8/14/14, 8:30 AM, "Kyle Mestery" >>> mailto:mest...@mestery.com>> wrote: >>> >>>> Folks, I'm not sure if all CI accounts are running sufficient tests. >>>> Per the requirements wiki page here [1], everyone needs to be running >>>> more than just Tempest API tests, which I still see most neutron >>>> third-party CI setups doing. I'd like to ask everyone who operates a >>>> third-party CI account for Neutron to please look at the link below >>>> and make sure you are running appropriate tests. If you have >>>> questions, the weekly third-party meeting [2] is a great place to ask >>>> questions. >>>> >>>> Thanks, >>>> Kyle >>>> >>>> [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>>> [2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>>>
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Parikshit, I will add your information right now. Thanks, Edgar On 8/19/14, 4:11 AM, "Parikshit Manur" wrote: >Hi Edgar, > NetScaler CI is not listed in the report. NetScaler CI up and running >and it is voting for patchsets which contains LBaaS changes, that's why >you did not see it voting for the test change ref. > Also, you can add me as a contact person and alias >networking-cloudinteg...@citrix.com as email contact point for the >NetScaler CI. > > Please let me know if you have any questions. > >Thanks, >Parikshit Manur > >-Original Message- >From: Edgar Magana [mailto:edgar.mag...@workday.com] >Sent: Saturday, 16 August 2014 4:06 AM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are >required to be run >Importance: High > >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore >this report that plugin/driver will be candidate to be remove from >Neutron tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below and >>make sure you are running appropriate tests. If you have questions, the >>weekly third-party meeting [2] is a great place to ask questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, NetScaler CI is not listed in the report. NetScaler CI up and running and it is voting for patchsets which contains LBaaS changes, that's why you did not see it voting for the test change ref. Also, you can add me as a contact person and alias networking-cloudinteg...@citrix.com as email contact point for the NetScaler CI. Please let me know if you have any questions. Thanks, Parikshit Manur -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Saturday, 16 August 2014 4:06 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Importance: High Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below and >make sure you are running appropriate tests. If you have questions, the >weekly third-party meeting [2] is a great place to ask questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Salvatore,
Thank you for your input. I actually took your suggestion about the area to
cover for the CI systems and I updated the wiki page for the third party
testing in Neutron.
https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
Cheers,
Edgar
From: Salvatore Orlando mailto:sorla...@nicira.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
mailto:openstack-dev@lists.openstack.org>>
Date: Friday, August 15, 2014 at 4:11 PM
To: "OpenStack Development Mailing List (not for usage questions)"
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to
be run
VMware minesweeper has filters which have been designed to cover the largest
possible subset of submissions without add unnecessary load to our scarce
resources for CI validation. This is probably why the analysis reveals not all
patches are covered.
Therefore our filters exclude neutral changes such as those in README.rst
(they're unlikely to crash neutron are they?), or changes in wsgi.py. The
latter is because the wsgi framework is shared with all plugins, and when it
breaks the NSX plugin it's very likely upstream tests will break too.
I think the areas which are important to cover are:
- neutron/agents/.* (or at least the agents you use)
- neutron/openstack/common/.*
- neutron/notifiers/.* (if your drivers report vif plugging events to nova)
- neutron/db/.*
- neutron/services/${thepluginsyouruninyourci_eveniftheyrenotyours}
- your plugin patch
Salvatore
PS: Openstack proposal bot submission's checks have been suspended for vmware
minesweeper about 2 days ago. They will resume soon.
On 16 August 2014 01:02, Edgar Magana
mailto:edgar.mag...@workday.com>> wrote:
Ivar,
Very valid point. This is why I need clarification from those CI owner.
I will run a new test with a basic change in the Neutron DB code. That should
be covered by almost all CI systems.
Edgar
From: Ivar Lazzaro mailto:ivarlazz...@gmail.com>>
Reply-To: "OpenStack Development Mailing List (not for usage questions)"
mailto:openstack-dev@lists.openstack.org>>
Date: Friday, August 15, 2014 at 3:47 PM
To: "OpenStack Development Mailing List (not for usage questions)"
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to
be run
Hi Edgar,
Nice shot, to be the inquisitor is not necessarily a bad thing :)
I know some CIs are 'stuck' waiting for bugs to be fixed or certain patches to
be merged, but I was wondering if it is a requirement that CIs vote *ALL* the
Neutron patches. Some may have missed your call just because of their filters
(see [0] section 'what changes to run against').
Cheers,
Ivar.
[0] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
On Sat, Aug 16, 2014 at 12:35 AM, Edgar Magana
mailto:edgar.mag...@workday.com>> wrote:
Team,
I did a quick audit on the Neutron CI. Very sad results. Only few plugins
and drivers are running properly and testing all Neutron commits.
I created a report here:
https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin
_and_Drivers
We will discuss the actions to take on the next Neutron IRC meeting. So
please, reach me out to clarify what is the status of your CI.
I had two commits to quickly verify the CI reliability:
https://review.openstack.org/#/c/114393/
https://review.openstack.org/#/c/40296/
I would expect all plugins and drivers passing on the first one and
failing for the second but I got so many surprises.
Neutron code quality and reliability is a top priority, if you ignore this
report that plugin/driver will be candidate to be remove from Neutron tree.
Cheers,
Edgar
P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job!
On 8/14/14, 8:30 AM, "Kyle Mestery"
mailto:mest...@mestery.com>> wrote:
>Folks, I'm not sure if all CI accounts are running sufficient tests.
>Per the requirements wiki page here [1], everyone needs to be running
>more than just Tempest API tests, which I still see most neutron
>third-party CI setups doing. I'd like to ask everyone who operates a
>third-party CI account for Neutron to please look at the link below
>and make sure you are running appropriate tests. If you have
>questions, the weekly third-party meeting [2] is a great place to ask
>questions.
>
>Thanks,
>Kyle
>
>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hello Folks, Based on today’s Neutron IRC meeting. I have modified the following wiki: https://wiki.openstack.org/wiki/NeutronThirdPartyTesting You will find a new suction with the minimal requirements for Juno. If you have still some questions, please contact me directly. I will start contacting each one of the owner directly and will update the table with the current Plugins and Drivers: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers Kind Regards, Edgar From: Hemanth Ravi mailto:hemanthrav...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Monday, August 18, 2014 at 12:24 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>>, Kedar K mailto:kedar.kulka...@oneconvergence.com>>, Deepak Gupta mailto:deepak.gu...@oneconvergence.com>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar, Our CI is running the tests (non voting), but don't see it listed on the review for any patch. Is this due to missing logs? I would like to confirm this is the issue, will resolve this. Thanks, -hemanth On Mon, Aug 18, 2014 at 8:35 AM, Edgar Magana mailto:edgar.mag...@workday.com>> wrote: Thank you Akihiro. I will propose a better organization for this section. Stay tune! Edgar On 8/17/14, 10:53 PM, "Akihiro Motoki" mailto:mot...@da.jp.nec.com>> wrote: > >On 2014/08/18 0:12, Kyle Mestery wrote: >> On Fri, Aug 15, 2014 at 5:35 PM, Edgar Magana >>mailto:edgar.mag...@workday.com>> wrote: >>> Team, >>> >>> I did a quick audit on the Neutron CI. Very sad results. Only few >>>plugins >>> and drivers are running properly and testing all Neutron commits. >>> I created a report here: >>> >>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plu >>>gin >>> _and_Drivers >>> >> Can you link this and/or move it to this page: >> >> https://wiki.openstack.org/wiki/NeutronPlugins >> >> This is under the "NeutronPolicies" wiki page which I did at the start >> of Juno. This tracks all policies and procedures for Neutron, and >> there's a Plugins page (which I linked to above) where this should >> land. > >I just added the link Neutron_Plugins_and_Drivers#Existing_Plugin >to NeutronPlugins wiki. > >The wiki pages NeutronPlugins and Neutron_Plugins_and_Drivers >cover the similar contents. According to the history of the page, >the latter one was created by Mark at Nov 2013 (beginning of Icehouse >cycle). >It seems better to merge these two pages to avoid the confusion. > >Akihiro > >> >>> >>> We will discuss the actions to take on the next Neutron IRC meeting. So >>> please, reach me out to clarify what is the status of your CI. >>> I had two commits to quickly verify the CI reliability: >>> >>> https://review.openstack.org/#/c/114393/ >>> >>> https://review.openstack.org/#/c/40296/ >>> >>> >>> I would expect all plugins and drivers passing on the first one and >>> failing for the second but I got so many surprises. >>> >>> Neutron code quality and reliability is a top priority, if you ignore >>>this >>> report that plugin/driver will be candidate to be remove from Neutron >>>tree. >>> >>> Cheers, >>> >>> Edgar >>> >>> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>>job! >>> >> Thanks for sending this out Edgar and doing this analysis! Can you >> please put an agenda item on Monday's meeting to discuss this? I won't >> be at the meeting as I'm on PTO (Mark is running the meeting in my >> absence), but I'd like the team to discuss this and allow all >> third-party people a chance to be there and share their feelings here. >> >> Thanks, >> Kyle >> >>> >>> On 8/14/14, 8:30 AM, "Kyle Mestery" >>> mailto:mest...@mestery.com>> wrote: >>> >>>> Folks, I'm not sure if all CI accounts are running sufficient tests. >>>> Per the requirements wiki page here [1], everyone needs to be running >>>> more than just Tempest API tests, which I still see most neutron >>>> third-party CI setups doing. I'd like to ask everyone who operates a >>>> third-party CI account for Neutron to please look at the link below >>>> and make sure you are running appropriate tests. If you
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Edgar, Our CI is running the tests (non voting), but don't see it listed on the review for any patch. Is this due to missing logs? I would like to confirm this is the issue, will resolve this. Thanks, -hemanth On Mon, Aug 18, 2014 at 8:35 AM, Edgar Magana wrote: > Thank you Akihiro. > > I will propose a better organization for this section. Stay tune! > > Edgar > > On 8/17/14, 10:53 PM, "Akihiro Motoki" wrote: > > > > >On 2014/08/18 0:12, Kyle Mestery wrote: > >> On Fri, Aug 15, 2014 at 5:35 PM, Edgar Magana > >> wrote: > >>> Team, > >>> > >>> I did a quick audit on the Neutron CI. Very sad results. Only few > >>>plugins > >>> and drivers are running properly and testing all Neutron commits. > >>> I created a report here: > >>> > >>> > https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plu > >>>gin > >>> _and_Drivers > >>> > >> Can you link this and/or move it to this page: > >> > >> https://wiki.openstack.org/wiki/NeutronPlugins > >> > >> This is under the "NeutronPolicies" wiki page which I did at the start > >> of Juno. This tracks all policies and procedures for Neutron, and > >> there's a Plugins page (which I linked to above) where this should > >> land. > > > >I just added the link Neutron_Plugins_and_Drivers#Existing_Plugin > >to NeutronPlugins wiki. > > > >The wiki pages NeutronPlugins and Neutron_Plugins_and_Drivers > >cover the similar contents. According to the history of the page, > >the latter one was created by Mark at Nov 2013 (beginning of Icehouse > >cycle). > >It seems better to merge these two pages to avoid the confusion. > > > >Akihiro > > > >> > >>> > >>> We will discuss the actions to take on the next Neutron IRC meeting. So > >>> please, reach me out to clarify what is the status of your CI. > >>> I had two commits to quickly verify the CI reliability: > >>> > >>> https://review.openstack.org/#/c/114393/ > >>> > >>> https://review.openstack.org/#/c/40296/ > >>> > >>> > >>> I would expect all plugins and drivers passing on the first one and > >>> failing for the second but I got so many surprises. > >>> > >>> Neutron code quality and reliability is a top priority, if you ignore > >>>this > >>> report that plugin/driver will be candidate to be remove from Neutron > >>>tree. > >>> > >>> Cheers, > >>> > >>> Edgar > >>> > >>> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty > >>>job! > >>> > >> Thanks for sending this out Edgar and doing this analysis! Can you > >> please put an agenda item on Monday's meeting to discuss this? I won't > >> be at the meeting as I'm on PTO (Mark is running the meeting in my > >> absence), but I'd like the team to discuss this and allow all > >> third-party people a chance to be there and share their feelings here. > >> > >> Thanks, > >> Kyle > >> > >>> > >>> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>> > Folks, I'm not sure if all CI accounts are running sufficient tests. > Per the requirements wiki page here [1], everyone needs to be running > more than just Tempest API tests, which I still see most neutron > third-party CI setups doing. I'd like to ask everyone who operates a > third-party CI account for Neutron to please look at the link below > and make sure you are running appropriate tests. If you have > questions, the weekly third-party meeting [2] is a great place to ask > questions. > > Thanks, > Kyle > > [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > [2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>> > >>> > >>> ___ > >>> OpenStack-dev mailing list > >>> OpenStack-dev@lists.openstack.org > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >> ___ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > >___ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Thank you Akihiro. I will propose a better organization for this section. Stay tune! Edgar On 8/17/14, 10:53 PM, "Akihiro Motoki" wrote: > >On 2014/08/18 0:12, Kyle Mestery wrote: >> On Fri, Aug 15, 2014 at 5:35 PM, Edgar Magana >> wrote: >>> Team, >>> >>> I did a quick audit on the Neutron CI. Very sad results. Only few >>>plugins >>> and drivers are running properly and testing all Neutron commits. >>> I created a report here: >>> >>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plu >>>gin >>> _and_Drivers >>> >> Can you link this and/or move it to this page: >> >> https://wiki.openstack.org/wiki/NeutronPlugins >> >> This is under the "NeutronPolicies" wiki page which I did at the start >> of Juno. This tracks all policies and procedures for Neutron, and >> there's a Plugins page (which I linked to above) where this should >> land. > >I just added the link Neutron_Plugins_and_Drivers#Existing_Plugin >to NeutronPlugins wiki. > >The wiki pages NeutronPlugins and Neutron_Plugins_and_Drivers >cover the similar contents. According to the history of the page, >the latter one was created by Mark at Nov 2013 (beginning of Icehouse >cycle). >It seems better to merge these two pages to avoid the confusion. > >Akihiro > >> >>> >>> We will discuss the actions to take on the next Neutron IRC meeting. So >>> please, reach me out to clarify what is the status of your CI. >>> I had two commits to quickly verify the CI reliability: >>> >>> https://review.openstack.org/#/c/114393/ >>> >>> https://review.openstack.org/#/c/40296/ >>> >>> >>> I would expect all plugins and drivers passing on the first one and >>> failing for the second but I got so many surprises. >>> >>> Neutron code quality and reliability is a top priority, if you ignore >>>this >>> report that plugin/driver will be candidate to be remove from Neutron >>>tree. >>> >>> Cheers, >>> >>> Edgar >>> >>> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >>>job! >>> >> Thanks for sending this out Edgar and doing this analysis! Can you >> please put an agenda item on Monday's meeting to discuss this? I won't >> be at the meeting as I'm on PTO (Mark is running the meeting in my >> absence), but I'd like the team to discuss this and allow all >> third-party people a chance to be there and share their feelings here. >> >> Thanks, >> Kyle >> >>> >>> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >>> Folks, I'm not sure if all CI accounts are running sufficient tests. Per the requirements wiki page here [1], everyone needs to be running more than just Tempest API tests, which I still see most neutron third-party CI setups doing. I'd like to ask everyone who operates a third-party CI account for Neutron to please look at the link below and make sure you are running appropriate tests. If you have questions, the weekly third-party meeting [2] is a great place to ask questions. Thanks, Kyle [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting [2] https://wiki.openstack.org/wiki/Meetings/ThirdParty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Neutron Ci Folks, I have received answers from almost all the CI contacts and I want to thank you all. Every case is different and I will review each one of your answer and questions. I do understand that every CI is different and this is why I would suggest two things: 1) Today's Neutron IRC meeting we can discuss the current status and what we want to achieve by the end of Juno release. 2) Have a short session during the Kilo summit to have an agreement on the requirements for the plugins and drivers under Neutron tree. I will answer each one of you and again I thank you all for your responses. Thanks, Edgar On 8/15/14, 3:35 PM, "Edgar Magana" wrote: >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore this >report that plugin/driver will be candidate to be remove from Neutron >tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
On 2014/08/18 0:12, Kyle Mestery wrote: > On Fri, Aug 15, 2014 at 5:35 PM, Edgar Magana > wrote: >> Team, >> >> I did a quick audit on the Neutron CI. Very sad results. Only few plugins >> and drivers are running properly and testing all Neutron commits. >> I created a report here: >> https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin >> _and_Drivers >> > Can you link this and/or move it to this page: > > https://wiki.openstack.org/wiki/NeutronPlugins > > This is under the "NeutronPolicies" wiki page which I did at the start > of Juno. This tracks all policies and procedures for Neutron, and > there's a Plugins page (which I linked to above) where this should > land. I just added the link Neutron_Plugins_and_Drivers#Existing_Plugin to NeutronPlugins wiki. The wiki pages NeutronPlugins and Neutron_Plugins_and_Drivers cover the similar contents. According to the history of the page, the latter one was created by Mark at Nov 2013 (beginning of Icehouse cycle). It seems better to merge these two pages to avoid the confusion. Akihiro > >> >> We will discuss the actions to take on the next Neutron IRC meeting. So >> please, reach me out to clarify what is the status of your CI. >> I had two commits to quickly verify the CI reliability: >> >> https://review.openstack.org/#/c/114393/ >> >> https://review.openstack.org/#/c/40296/ >> >> >> I would expect all plugins and drivers passing on the first one and >> failing for the second but I got so many surprises. >> >> Neutron code quality and reliability is a top priority, if you ignore this >> report that plugin/driver will be candidate to be remove from Neutron tree. >> >> Cheers, >> >> Edgar >> >> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! >> > Thanks for sending this out Edgar and doing this analysis! Can you > please put an agenda item on Monday's meeting to discuss this? I won't > be at the meeting as I'm on PTO (Mark is running the meeting in my > absence), but I'd like the team to discuss this and allow all > third-party people a chance to be there and share their feelings here. > > Thanks, > Kyle > >> >> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >> >>> Folks, I'm not sure if all CI accounts are running sufficient tests. >>> Per the requirements wiki page here [1], everyone needs to be running >>> more than just Tempest API tests, which I still see most neutron >>> third-party CI setups doing. I'd like to ask everyone who operates a >>> third-party CI account for Neutron to please look at the link below >>> and make sure you are running appropriate tests. If you have >>> questions, the weekly third-party meeting [2] is a great place to ask >>> questions. >>> >>> Thanks, >>> Kyle >>> >>> [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>> [2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >>> >>> ___ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, Freescale CI is not listed in the below report: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Pl We are following all the requirements of CI setup and as well participating in the IRC Meeting. Can you please let us know if we are missing any other requirements of CI Setup. Regards, Balaji.P > -Original Message- > From: trinath.soman...@freescale.com > [mailto:trinath.soman...@freescale.com] > Sent: Sunday, August 17, 2014 12:04 PM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > Hi Edgar, > > Freescale CI is reporting the results for ML2 Mechanism driver (J-1) and > FWaaS Plugin (to be approved for J-3). > I'm the owner for this CI. > > The Wiki page for this CI is > https://wiki.openstack.org/wiki/ThirdPartySystems/Freescale_CI. > > -- > Trinath Somanchi - B39208 > trinath.soman...@freescale.com | extn: 4048 > > -Original Message- > From: Edgar Magana [mailto:edgar.mag...@workday.com] > Sent: Saturday, August 16, 2014 4:06 AM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > Importance: High > > Team, > > I did a quick audit on the Neutron CI. Very sad results. Only few plugins > and drivers are running properly and testing all Neutron commits. > I created a report here: > https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plug > in > _and_Drivers > > > We will discuss the actions to take on the next Neutron IRC meeting. So > please, reach me out to clarify what is the status of your CI. > I had two commits to quickly verify the CI reliability: > > https://review.openstack.org/#/c/114393/ > > https://review.openstack.org/#/c/40296/ > > > I would expect all plugins and drivers passing on the first one and > failing for the second but I got so many surprises. > > Neutron code quality and reliability is a top priority, if you ignore > this report that plugin/driver will be candidate to be remove from > Neutron tree. > > Cheers, > > Edgar > > P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty > job! > > > On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > > >Folks, I'm not sure if all CI accounts are running sufficient tests. > >Per the requirements wiki page here [1], everyone needs to be running > >more than just Tempest API tests, which I still see most neutron > >third-party CI setups doing. I'd like to ask everyone who operates a > >third-party CI account for Neutron to please look at the link below and > >make sure you are running appropriate tests. If you have questions, the > >weekly third-party meeting [2] is a great place to ask questions. > > > >Thanks, > >Kyle > > > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > > > >___ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Edgar: The Cisco APIC should be reporting results for both APIC-related and non-APIC related changes now. (See http://cisco-neutron-ci.cisco.com/logs/apic/1738/). Will you be updating the wiki page? -Dane -Original Message- From: Dane Leblanc (leblancd) Sent: Friday, August 15, 2014 8:18 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Also, you can add me as a contact person for the Cisco VPNaaS driver. -Original Message- From: Dane Leblanc (leblancd) Sent: Friday, August 15, 2014 8:14 PM To: OpenStack Development Mailing List (not for usage questions) Subject: RE: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar: For the Notes for the Cisco APIC, can you change the comment "results are fake" to something like "results are only valid for APIC-related commits"? I think this more accurately represents our current results (for reasons we chatted about on another thread). Thanks, Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Friday, August 15, 2014 6:36 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Importance: High Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below and >make sure you are running appropriate tests. If you have questions, the >weekly third-party meeting [2] is a great place to ask questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
On Fri, Aug 15, 2014 at 5:35 PM, Edgar Magana wrote: > Team, > > I did a quick audit on the Neutron CI. Very sad results. Only few plugins > and drivers are running properly and testing all Neutron commits. > I created a report here: > https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin > _and_Drivers > Can you link this and/or move it to this page: https://wiki.openstack.org/wiki/NeutronPlugins This is under the "NeutronPolicies" wiki page which I did at the start of Juno. This tracks all policies and procedures for Neutron, and there's a Plugins page (which I linked to above) where this should land. > > We will discuss the actions to take on the next Neutron IRC meeting. So > please, reach me out to clarify what is the status of your CI. > I had two commits to quickly verify the CI reliability: > > https://review.openstack.org/#/c/114393/ > > https://review.openstack.org/#/c/40296/ > > > I would expect all plugins and drivers passing on the first one and > failing for the second but I got so many surprises. > > Neutron code quality and reliability is a top priority, if you ignore this > report that plugin/driver will be candidate to be remove from Neutron tree. > > Cheers, > > Edgar > > P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! > Thanks for sending this out Edgar and doing this analysis! Can you please put an agenda item on Monday's meeting to discuss this? I won't be at the meeting as I'm on PTO (Mark is running the meeting in my absence), but I'd like the team to discuss this and allow all third-party people a chance to be there and share their feelings here. Thanks, Kyle > > On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, Freescale CI is reporting the results for ML2 Mechanism driver (J-1) and FWaaS Plugin (to be approved for J-3). I'm the owner for this CI. The Wiki page for this CI is https://wiki.openstack.org/wiki/ThirdPartySystems/Freescale_CI. -- Trinath Somanchi - B39208 trinath.soman...@freescale.com | extn: 4048 -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Saturday, August 16, 2014 4:06 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Importance: High Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below and >make sure you are running appropriate tests. If you have questions, the >weekly third-party meeting [2] is a great place to ask questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
wrong link- please take a look at this one after log fixes have been put in - Gerrit Code Review Gerrit Code Review Loading Gerrit Code Review ... Gerrit requires a JavaScript enabled browser. View on review.openstack.org Preview by Yahoo From: daya kamath To: OpenStack Development Mailing List (not for usage questions) Sent: Saturday, August 16, 2014 9:46 AM Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run hi edgar, the IBM SDNVE system runs a subset of tempest tests currently, but is triggered for all neutron changes. we are working on adding more tests to our system. There was an issue with uploading all the logs properly, which i have fixed. one pending item is to upload the console log as well. also, am experiencing some tempest failures since this morning, and i'm investigating those presently. please take a look at this run from today, for any additional feedback. thanks for your input. Gerrit Code Review Gerrit Code Review Loading Gerrit Code Review ... Gerrit requires a JavaScript enabled browser. View on review.openstack.org Preview by Yahoo thanks! daya kamath From: Edgar Magana To: OpenStack Development Mailing List (not for usage questions) Sent: Saturday, August 16, 2014 4:29 AM Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Correction The right links are: https://review.openstack.org/#/c/114629/ https://review.openstack.org/#/c/114393/ Not this one: https://review.openstack.org/#/c/40296/ Edgar On 8/15/14, 3:35 PM, "Edgar Magana" wrote: >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore this >report that plugin/driver will be candidate to be remove from Neutron >tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
hi edgar, the IBM SDNVE system runs a subset of tempest tests currently, but is triggered for all neutron changes. we are working on adding more tests to our system. There was an issue with uploading all the logs properly, which i have fixed. one pending item is to upload the console log as well. also, am experiencing some tempest failures since this morning, and i'm investigating those presently. please take a look at this run from today, for any additional feedback. thanks for your input. Gerrit Code Review Gerrit Code Review Loading Gerrit Code Review ... Gerrit requires a JavaScript enabled browser. View on review.openstack.org Preview by Yahoo thanks! daya kamath From: Edgar Magana To: OpenStack Development Mailing List (not for usage questions) Sent: Saturday, August 16, 2014 4:29 AM Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Correction The right links are: https://review.openstack.org/#/c/114629/ https://review.openstack.org/#/c/114393/ Not this one: https://review.openstack.org/#/c/40296/ Edgar On 8/15/14, 3:35 PM, "Edgar Magana" wrote: >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore this >report that plugin/driver will be candidate to be remove from Neutron >tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, Brocade Vyatta CI is reporting the results and providing log links. For Brocade Vyatta Plugin, I have updated my name as the owner. For Brocade VDX Plugin, Shiv Haris is the owner. Please let me know if you have any questions. Thanks, Karthik -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Friday, August 15, 2014 3:59 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Importance: High Correction The right links are: https://review.openstack.org/#/c/114629/ https://review.openstack.org/#/c/114393/ Not this one: https://review.openstack.org/#/c/40296/ Edgar On 8/15/14, 3:35 PM, "Edgar Magana" wrote: >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few >plugins and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Pl >ugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore >this report that plugin/driver will be candidate to be remove from >Neutron tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, I am the reference for the Embrane CI (added my name to the page you linked) Our Ci had some issues that were pointed out during a third party meeting some weeks ago, we solved the issues that were reported and the CI was running fine. There was a list of issues tracked on this etherpad for various CIs https://etherpad.openstack.org/p/neutron-thirdparty-juno Right now it's only running against changes to the plugin until this review gets finalized and merged: https://review.openstack.org/#/c/108226/ Ignacio -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Friday, August 15, 2014 3:59 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Importance: High Correction The right links are: https://review.openstack.org/#/c/114629/ https://review.openstack.org/#/c/114393/ Not this one: https://review.openstack.org/#/c/40296/ Edgar On 8/15/14, 3:35 PM, "Edgar Magana" wrote: >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few >plugins and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Pl >ugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore >this report that plugin/driver will be candidate to be remove from >Neutron tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi, Edgar MetapluginCI didn't run for document changes. I changed filter so that CI run for ALL changes include documents. Thanks, Hirofumi 2014-08-16 7:35 GMT+09:00 Edgar Magana : > Team, > > I did a quick audit on the Neutron CI. Very sad results. Only few plugins > and drivers are running properly and testing all Neutron commits. > I created a report here: > https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin > _and_Drivers > > > We will discuss the actions to take on the next Neutron IRC meeting. So > please, reach me out to clarify what is the status of your CI. > I had two commits to quickly verify the CI reliability: > > https://review.openstack.org/#/c/114393/ > > https://review.openstack.org/#/c/40296/ > > > I would expect all plugins and drivers passing on the first one and > failing for the second but I got so many surprises. > > Neutron code quality and reliability is a top priority, if you ignore this > report that plugin/driver will be candidate to be remove from Neutron tree. > > Cheers, > > Edgar > > P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! > > > On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > > >Folks, I'm not sure if all CI accounts are running sufficient tests. > >Per the requirements wiki page here [1], everyone needs to be running > >more than just Tempest API tests, which I still see most neutron > >third-party CI setups doing. I'd like to ask everyone who operates a > >third-party CI account for Neutron to please look at the link below > >and make sure you are running appropriate tests. If you have > >questions, the weekly third-party meeting [2] is a great place to ask > >questions. > > > >Thanks, > >Kyle > > > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > > > >___ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Also, you can add me as a contact person for the Cisco VPNaaS driver. -Original Message- From: Dane Leblanc (leblancd) Sent: Friday, August 15, 2014 8:14 PM To: OpenStack Development Mailing List (not for usage questions) Subject: RE: [openstack-dev] [neutron] [third-party] What tests are required to be run Edgar: For the Notes for the Cisco APIC, can you change the comment "results are fake" to something like "results are only valid for APIC-related commits"? I think this more accurately represents our current results (for reasons we chatted about on another thread). Thanks, Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Friday, August 15, 2014 6:36 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Importance: High Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below and >make sure you are running appropriate tests. If you have questions, the >weekly third-party meeting [2] is a great place to ask questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Edgar: For the Notes for the Cisco APIC, can you change the comment "results are fake" to something like "results are only valid for APIC-related commits"? I think this more accurately represents our current results (for reasons we chatted about on another thread). Thanks, Dane -Original Message- From: Edgar Magana [mailto:edgar.mag...@workday.com] Sent: Friday, August 15, 2014 6:36 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Importance: High Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below and >make sure you are running appropriate tests. If you have questions, the >weekly third-party meeting [2] is a great place to ask questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, I am running the Nuage CI. The Nuage CI has run and posted successfully the result for the first review. Because of infra issue(internet can be down, etc) we are running manually the -1 -1 take some time away for developper and before doing a -1, I want to be sure that it's a valid issue. (we had a talk about it during the third-party meeting on mondays) I have posted the failure report and you can see that it was run, uploaded the result to our server but we didn't vote (reason mention above) http://208.113.169.228/nuage-ci/29_114629_1/ (seetimestamp of the files) Let me know if you have any questions. Franck (lyxus) Franck On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana wrote: > Team, > > I did a quick audit on the Neutron CI. Very sad results. Only few plugins > and drivers are running properly and testing all Neutron commits. > I created a report here: > https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin > _and_Drivers > > > We will discuss the actions to take on the next Neutron IRC meeting. So > please, reach me out to clarify what is the status of your CI. > I had two commits to quickly verify the CI reliability: > > https://review.openstack.org/#/c/114393/ > > https://review.openstack.org/#/c/40296/ > > > I would expect all plugins and drivers passing on the first one and > failing for the second but I got so many surprises. > > Neutron code quality and reliability is a top priority, if you ignore this > report that plugin/driver will be candidate to be remove from Neutron tree. > > Cheers, > > Edgar > > P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! > > > On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > > >Folks, I'm not sure if all CI accounts are running sufficient tests. > >Per the requirements wiki page here [1], everyone needs to be running > >more than just Tempest API tests, which I still see most neutron > >third-party CI setups doing. I'd like to ask everyone who operates a > >third-party CI account for Neutron to please look at the link below > >and make sure you are running appropriate tests. If you have > >questions, the weekly third-party meeting [2] is a great place to ask > >questions. > > > >Thanks, > >Kyle > > > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > > > >___ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
You didn't wait long enough for the Big Switch CI to reach your negative test. :-) Currently the CI system is backed by about 100 patches to push responses out ~22 hours. This does bring up an old discussion that was never resolved. What is the minimum expected response time for CI systems? On Fri, Aug 15, 2014 at 3:35 PM, Edgar Magana wrote: > Team, > > I did a quick audit on the Neutron CI. Very sad results. Only few plugins > and drivers are running properly and testing all Neutron commits. > I created a report here: > https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin > _and_Drivers > > > We will discuss the actions to take on the next Neutron IRC meeting. So > please, reach me out to clarify what is the status of your CI. > I had two commits to quickly verify the CI reliability: > > https://review.openstack.org/#/c/114393/ > > https://review.openstack.org/#/c/40296/ > > > I would expect all plugins and drivers passing on the first one and > failing for the second but I got so many surprises. > > Neutron code quality and reliability is a top priority, if you ignore this > report that plugin/driver will be candidate to be remove from Neutron tree. > > Cheers, > > Edgar > > P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! > > > On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > > >Folks, I'm not sure if all CI accounts are running sufficient tests. > >Per the requirements wiki page here [1], everyone needs to be running > >more than just Tempest API tests, which I still see most neutron > >third-party CI setups doing. I'd like to ask everyone who operates a > >third-party CI account for Neutron to please look at the link below > >and make sure you are running appropriate tests. If you have > >questions, the weekly third-party meeting [2] is a great place to ask > >questions. > > > >Thanks, > >Kyle > > > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > > > >___ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Kevin Benton ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
That's great! Also, I would suggest we decide on a 'standard' way for CI owners to comunicate their CIs status (eg. MyCompany CI is not voting waiting for bug #9001 to be fixed). Using the ML for that may be confusing... What about an etherpad (linked from [0]) or a dedicated section of [0] itself? Ivar. [0] https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin_and_Drivers On Sat, Aug 16, 2014 at 1:02 AM, Edgar Magana wrote: > Ivar, > > Very valid point. This is why I need clarification from those CI owner. > I will run a new test with a basic change in the Neutron DB code. That > should be covered by almost all CI systems. > > Edgar > > From: Ivar Lazzaro > Reply-To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Date: Friday, August 15, 2014 at 3:47 PM > To: "OpenStack Development Mailing List (not for usage questions)" < > openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [neutron] [third-party] What tests are > required to be run > > Hi Edgar, > > Nice shot, to be the inquisitor is not necessarily a bad thing :) > > I know some CIs are 'stuck' waiting for bugs to be fixed or certain > patches to be merged, but I was wondering if it is a requirement that CIs > vote *ALL* the Neutron patches. Some may have missed your call just because > of their filters (see [0] section 'what changes to run against'). > > Cheers, > Ivar. > > [0] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > > > > > On Sat, Aug 16, 2014 at 12:35 AM, Edgar Magana > wrote: > >> Team, >> >> I did a quick audit on the Neutron CI. Very sad results. Only few plugins >> and drivers are running properly and testing all Neutron commits. >> I created a report here: >> >> https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin >> _and_Drivers >> >> >> We will discuss the actions to take on the next Neutron IRC meeting. So >> please, reach me out to clarify what is the status of your CI. >> I had two commits to quickly verify the CI reliability: >> >> https://review.openstack.org/#/c/114393/ >> >> https://review.openstack.org/#/c/40296/ >> >> >> I would expect all plugins and drivers passing on the first one and >> failing for the second but I got so many surprises. >> >> Neutron code quality and reliability is a top priority, if you ignore this >> report that plugin/driver will be candidate to be remove from Neutron >> tree. >> >> Cheers, >> >> Edgar >> >> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >> job! >> >> >> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >> >> >Folks, I'm not sure if all CI accounts are running sufficient tests. >> >Per the requirements wiki page here [1], everyone needs to be running >> >more than just Tempest API tests, which I still see most neutron >> >third-party CI setups doing. I'd like to ask everyone who operates a >> >third-party CI account for Neutron to please look at the link below >> >and make sure you are running appropriate tests. If you have >> >questions, the weekly third-party meeting [2] is a great place to ask >> >questions. >> > >> >Thanks, >> >Kyle >> > >> >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >> >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> > >> >___ >> >OpenStack-dev mailing list >> >OpenStack-dev@lists.openstack.org >> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
VMware minesweeper has filters which have been designed to cover the
largest possible subset of submissions without add unnecessary load to our
scarce resources for CI validation. This is probably why the analysis
reveals not all patches are covered.
Therefore our filters exclude neutral changes such as those in README.rst
(they're unlikely to crash neutron are they?), or changes in wsgi.py. The
latter is because the wsgi framework is shared with all plugins, and when
it breaks the NSX plugin it's very likely upstream tests will break too.
I think the areas which are important to cover are:
- neutron/agents/.* (or at least the agents you use)
- neutron/openstack/common/.*
- neutron/notifiers/.* (if your drivers report vif plugging events to nova)
- neutron/db/.*
- neutron/services/${thepluginsyouruninyourci_eveniftheyrenotyours}
- your plugin patch
Salvatore
PS: Openstack proposal bot submission's checks have been suspended for
vmware minesweeper about 2 days ago. They will resume soon.
On 16 August 2014 01:02, Edgar Magana wrote:
> Ivar,
>
> Very valid point. This is why I need clarification from those CI owner.
> I will run a new test with a basic change in the Neutron DB code. That
> should be covered by almost all CI systems.
>
> Edgar
>
> From: Ivar Lazzaro
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Friday, August 15, 2014 at 3:47 PM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> required to be run
>
> Hi Edgar,
>
> Nice shot, to be the inquisitor is not necessarily a bad thing :)
>
> I know some CIs are 'stuck' waiting for bugs to be fixed or certain
> patches to be merged, but I was wondering if it is a requirement that CIs
> vote *ALL* the Neutron patches. Some may have missed your call just because
> of their filters (see [0] section 'what changes to run against').
>
> Cheers,
> Ivar.
>
> [0] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
>
>
>
>
> On Sat, Aug 16, 2014 at 12:35 AM, Edgar Magana
> wrote:
>
>> Team,
>>
>> I did a quick audit on the Neutron CI. Very sad results. Only few plugins
>> and drivers are running properly and testing all Neutron commits.
>> I created a report here:
>>
>> https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin
>> _and_Drivers
>>
>>
>> We will discuss the actions to take on the next Neutron IRC meeting. So
>> please, reach me out to clarify what is the status of your CI.
>> I had two commits to quickly verify the CI reliability:
>>
>> https://review.openstack.org/#/c/114393/
>>
>> https://review.openstack.org/#/c/40296/
>>
>>
>> I would expect all plugins and drivers passing on the first one and
>> failing for the second but I got so many surprises.
>>
>> Neutron code quality and reliability is a top priority, if you ignore this
>> report that plugin/driver will be candidate to be remove from Neutron
>> tree.
>>
>> Cheers,
>>
>> Edgar
>>
>> P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty
>> job!
>>
>>
>> On 8/14/14, 8:30 AM, "Kyle Mestery" wrote:
>>
>> >Folks, I'm not sure if all CI accounts are running sufficient tests.
>> >Per the requirements wiki page here [1], everyone needs to be running
>> >more than just Tempest API tests, which I still see most neutron
>> >third-party CI setups doing. I'd like to ask everyone who operates a
>> >third-party CI account for Neutron to please look at the link below
>> >and make sure you are running appropriate tests. If you have
>> >questions, the weekly third-party meeting [2] is a great place to ask
>> >questions.
>> >
>> >Thanks,
>> >Kyle
>> >
>> >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
>> >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
>> >
>> >___
>> >OpenStack-dev mailing list
>> >OpenStack-dev@lists.openstack.org
>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Ivar, Very valid point. This is why I need clarification from those CI owner. I will run a new test with a basic change in the Neutron DB code. That should be covered by almost all CI systems. Edgar From: Ivar Lazzaro mailto:ivarlazz...@gmail.com>> Reply-To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Date: Friday, August 15, 2014 at 3:47 PM To: "OpenStack Development Mailing List (not for usage questions)" mailto:openstack-dev@lists.openstack.org>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run Hi Edgar, Nice shot, to be the inquisitor is not necessarily a bad thing :) I know some CIs are 'stuck' waiting for bugs to be fixed or certain patches to be merged, but I was wondering if it is a requirement that CIs vote *ALL* the Neutron patches. Some may have missed your call just because of their filters (see [0] section 'what changes to run against'). Cheers, Ivar. [0] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On Sat, Aug 16, 2014 at 12:35 AM, Edgar Magana mailto:edgar.mag...@workday.com>> wrote: Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" mailto:mest...@mestery.com>> wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below >and make sure you are running appropriate tests. If you have >questions, the weekly third-party meeting [2] is a great place to ask >questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Correction The right links are: https://review.openstack.org/#/c/114629/ https://review.openstack.org/#/c/114393/ Not this one: https://review.openstack.org/#/c/40296/ Edgar On 8/15/14, 3:35 PM, "Edgar Magana" wrote: >Team, > >I did a quick audit on the Neutron CI. Very sad results. Only few plugins >and drivers are running properly and testing all Neutron commits. >I created a report here: >https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugi >n >_and_Drivers > > >We will discuss the actions to take on the next Neutron IRC meeting. So >please, reach me out to clarify what is the status of your CI. >I had two commits to quickly verify the CI reliability: > >https://review.openstack.org/#/c/114393/ > >https://review.openstack.org/#/c/40296/ > > >I would expect all plugins and drivers passing on the first one and >failing for the second but I got so many surprises. > >Neutron code quality and reliability is a top priority, if you ignore this >report that plugin/driver will be candidate to be remove from Neutron >tree. > >Cheers, > >Edgar > >P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty >job! > > >On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > >>Folks, I'm not sure if all CI accounts are running sufficient tests. >>Per the requirements wiki page here [1], everyone needs to be running >>more than just Tempest API tests, which I still see most neutron >>third-party CI setups doing. I'd like to ask everyone who operates a >>third-party CI account for Neutron to please look at the link below >>and make sure you are running appropriate tests. If you have >>questions, the weekly third-party meeting [2] is a great place to ask >>questions. >> >>Thanks, >>Kyle >> >>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty >> >>___ >>OpenStack-dev mailing list >>OpenStack-dev@lists.openstack.org >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi Edgar, Nice shot, to be the inquisitor is not necessarily a bad thing :) I know some CIs are 'stuck' waiting for bugs to be fixed or certain patches to be merged, but I was wondering if it is a requirement that CIs vote *ALL* the Neutron patches. Some may have missed your call just because of their filters (see [0] section 'what changes to run against'). Cheers, Ivar. [0] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting On Sat, Aug 16, 2014 at 12:35 AM, Edgar Magana wrote: > Team, > > I did a quick audit on the Neutron CI. Very sad results. Only few plugins > and drivers are running properly and testing all Neutron commits. > I created a report here: > https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin > _and_Drivers > > > We will discuss the actions to take on the next Neutron IRC meeting. So > please, reach me out to clarify what is the status of your CI. > I had two commits to quickly verify the CI reliability: > > https://review.openstack.org/#/c/114393/ > > https://review.openstack.org/#/c/40296/ > > > I would expect all plugins and drivers passing on the first one and > failing for the second but I got so many surprises. > > Neutron code quality and reliability is a top priority, if you ignore this > report that plugin/driver will be candidate to be remove from Neutron tree. > > Cheers, > > Edgar > > P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! > > > On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: > > >Folks, I'm not sure if all CI accounts are running sufficient tests. > >Per the requirements wiki page here [1], everyone needs to be running > >more than just Tempest API tests, which I still see most neutron > >third-party CI setups doing. I'd like to ask everyone who operates a > >third-party CI account for Neutron to please look at the link below > >and make sure you are running appropriate tests. If you have > >questions, the weekly third-party meeting [2] is a great place to ask > >questions. > > > >Thanks, > >Kyle > > > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting > >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > > > >___ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Team, I did a quick audit on the Neutron CI. Very sad results. Only few plugins and drivers are running properly and testing all Neutron commits. I created a report here: https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing_Plugin _and_Drivers We will discuss the actions to take on the next Neutron IRC meeting. So please, reach me out to clarify what is the status of your CI. I had two commits to quickly verify the CI reliability: https://review.openstack.org/#/c/114393/ https://review.openstack.org/#/c/40296/ I would expect all plugins and drivers passing on the first one and failing for the second but I got so many surprises. Neutron code quality and reliability is a top priority, if you ignore this report that plugin/driver will be candidate to be remove from Neutron tree. Cheers, Edgar P.s. I hate to be the inquisitor hereŠ but someone has to do the dirty job! On 8/14/14, 8:30 AM, "Kyle Mestery" wrote: >Folks, I'm not sure if all CI accounts are running sufficient tests. >Per the requirements wiki page here [1], everyone needs to be running >more than just Tempest API tests, which I still see most neutron >third-party CI setups doing. I'd like to ask everyone who operates a >third-party CI account for Neutron to please look at the link below >and make sure you are running appropriate tests. If you have >questions, the weekly third-party meeting [2] is a great place to ask >questions. > >Thanks, >Kyle > >[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting >[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty > >___ >OpenStack-dev mailing list >OpenStack-dev@lists.openstack.org >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Hi- I'm hitting bugs for (basic/advanced)_server_ops testing. https://bugs.launchpad.net/nova/+bug/1232303 Kindly help me with a fix to this. Thanking you. -- Trinath Somanchi - B39208 trinath.soman...@freescale.com | extn: 4048 -Original Message- From: Kyle Mestery [mailto:mest...@mestery.com] Sent: Thursday, August 14, 2014 9:00 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [neutron] [third-party] What tests are required to be run Folks, I'm not sure if all CI accounts are running sufficient tests. Per the requirements wiki page here [1], everyone needs to be running more than just Tempest API tests, which I still see most neutron third-party CI setups doing. I'd like to ask everyone who operates a third-party CI account for Neutron to please look at the link below and make sure you are running appropriate tests. If you have questions, the weekly third-party meeting [2] is a great place to ask questions. Thanks, Kyle [1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting [2] https://wiki.openstack.org/wiki/Meetings/ThirdParty ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
