Thanks Edgar for updating the APIC status!!! 

Edgar and Kyle: *****PLEASE NOTE******!!!!  I need your understanding and 
advice on the following:

We are still stuck with a problem stemming from a design limitation of Jenkins 
that prevents us from being compliant with Neutron 3rd Party CI requirements 
for our DFA CI.

The issue is that Jenkins only allows our scripts to (programmatically) return 
either Success or Fail. There is no option to return "Aborted", "Not Tested", 
or "Skipped".

Why does this matter? The DFA plugin is just being introduced, and initial 
DFA-enabling change sets have not yet been merged. Therefore, all other change 
sets will fail our Tempest tests, since they are not DFA-enabled.

Similarly, we were recently blocked in our APIC CI with a critical bug, causing 
all change sets without this fix to fail on our APIC testbed.

In these cases, we would like to enter a "throttled" or "partially blocked" 
mode, where we would skip testing on change sets we know will fail, and (in an 
ideal world) signal this shortcoming to Gerrit e.g. by returning a "Skipped" 
status. Unfortunately, this option is not available in Jenkins scripts, as 
Jenkins is currently designed. The only options we have available is "Success" 
or all "Fail", which are both misleading. We would also incorrectly report 
success or fail on one of the following test commits:
https://review.openstack.org/#/c/114393/
https://review.openstack.org/#/c/40296/

I've brought this issue up on the openstack-infra IRC, and jeblair confirmed 
the Jenkins limitation, but asked me to get consensus from the Neutron 
community as to this being a problem/requirement. I've also sent out an e-mail 
on the Neutron ML trying to start a discussion on this problem (no traction). I 
plan on bringing this up in the 3rd Party CI IRC on Monday, assuming there is 
time permitted in the open discussion.

I'm also investigating 

For the short term, I would like to propose the following:
* We bring this up on the 3rd Party CI IRC on Monday to get a solution or 
workaround, if available. If a solution is available, let's consider including 
that as a hint when we come up with CI requirements for handling CIs bocked by 
some critical fix.
* I'm also looking into using a REST API to cancel a Jenkins job 
programmatically.
* If no solution or workaround is available, we work with infra team or with 
Jenkins team to create a solution.
* Until a solution is available, for plugins which are blocked by a critical 
bug, we post a status/notes indicating the plugin's situation on our 3rd party 
CI status wiki, e.g.:

Vendor                  Plugin/Driver Name      Contact Name            Status  
Notes
My Vendor Name  My Plugin CI            My Contact Person       T       
Throttled / Partially blocked / Awaiting Intial Commits

The status/notes should be clear and understood by the Neutron team.  The 
console logs for change sets where the tests were skipped should also contain a 
message that all testing is being skipped for that commit.

Note that when the DFA initial commits are merged, then this issue would go 
away for the DFA CI. However, this problem will reappear every time a blocking 
critical bug shows up for a 3rd party CI setup, or a new plugin is introduced 
and the hardware-enabling commits are not yet merged.  (That is, until we have 
a solution for the Jenkins limitation).

Let me know what you think.

Thanks,
Dane

-----Original Message-----
From: Edgar Magana [mailto:edgar.mag...@workday.com] 
Sent: Friday, August 22, 2014 1:57 PM
To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage 
questions)
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to 
be run

Sorry my bad but I just changed.

Edgar

On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> wrote:

>Edgar:
>
>I'm still seeing the comment "Results are not accurate. Needs 
>clarification..."
>
>Dane
>
>-----Original Message-----
>From: Edgar Magana [mailto:edgar.mag...@workday.com]
>Sent: Thursday, August 21, 2014 2:58 PM
>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not 
>for usage questions)
>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>required to be run
>
>Dane,
>
>Wiki has been updated.
>
>Thanks,
>
>Edgar
>
>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com> wrote:
>
>>Edgar:
>>
>>The status on the wiki page says "Results are not accurate. Needs 
>>clarification from Cisco".
>>Can you please tell me what we are missing?
>>
>>-Dane
>>
>>-----Original Message-----
>>From: Dane Leblanc (leblancd)
>>Sent: Tuesday, August 19, 2014 3:05 PM
>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage
>>questions)
>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are 
>>required to be run
>>
>>The APIC CI did run tests against that commit (after some queue latency):
>>
>>http://128.107.233.28:8080/job/apic/1860/
>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/
>>
>>But the review comments never showed up on Gerrit. This seems to be an 
>>intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from 
>>this Jenkins/Gerrit server. Whenever we disable another one of our 
>>other Jenkins jobs (in this case, we disabled DFA for some rework), 
>>the review comments sometimes stop showing up on Gerrit.
>>
>>-----Original Message-----
>>From: Edgar Magana [mailto:edgar.mag...@workday.com]
>>Sent: Tuesday, August 19, 2014 1:33 PM
>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not 
>>for usage questions)
>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>required to be run
>>
>>I was looking to one of the most recent Neutron commits:
>>https://review.openstack.org/#/c/115175/
>>
>>
>>I could not find the APIC report.
>>
>>Edgar
>>
>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <lebla...@cisco.com>
>>wrote:
>>
>>>From which commit is it missing?
>>>https://review.openstack.org/#/c/114629/
>>>https://review.openstack.org/#/c/114393/
>>>
>>>-----Original Message-----
>>>From: Edgar Magana [mailto:edgar.mag...@workday.com]
>>>Sent: Tuesday, August 19, 2014 12:28 PM
>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not 
>>>for usage questions)
>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>>required to be run
>>>
>>>Dane,
>>>
>>>Are you sure about it?
>>>I just went to this commit and I could not find the APIC tests.
>>>
>>>Thanks,
>>>
>>>Edgar
>>>
>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <lebla...@cisco.com>
>>>wrote:
>>>
>>>>Edgar:
>>>>
>>>>The Cisco APIC should be reporting results for both APIC-related and 
>>>>non-APIC related changes now.
>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/).
>>>>
>>>>Will you be updating the wiki page?
>>>>
>>>>-Dane
>>>>
>>>>-----Original Message-----
>>>>From: Dane Leblanc (leblancd)
>>>>Sent: Friday, August 15, 2014 8:18 PM
>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>>>required to be run
>>>>
>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver.
>>>>
>>>>-----Original Message-----
>>>>From: Dane Leblanc (leblancd)
>>>>Sent: Friday, August 15, 2014 8:14 PM
>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are 
>>>>required to be run
>>>>
>>>>Edgar:
>>>>
>>>>For the Notes for the Cisco APIC, can you change the comment 
>>>>"results are fake" to something like "results are only valid for 
>>>>APIC-related commits"? I think this more accurately represents our 
>>>>current results (for reasons we chatted about on another thread).
>>>>
>>>>Thanks,
>>>>Dane
>>>>
>>>>-----Original Message-----
>>>>From: Edgar Magana [mailto:edgar.mag...@workday.com]
>>>>Sent: Friday, August 15, 2014 6:36 PM
>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>>>required to be run
>>>>Importance: High
>>>>
>>>>Team,
>>>>
>>>>I did a quick audit on the Neutron CI. Very sad results. Only few 
>>>>plugins and drivers are running properly and testing all Neutron 
>>>>commits.
>>>>I created a report here:
>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing
>>>>_
>>>>P
>>>>l
>>>>ugi
>>>>n
>>>>_and_Drivers
>>>>
>>>>
>>>>We will discuss the actions to take on the next Neutron IRC meeting.
>>>>So please, reach me out to clarify what is the status of your CI.
>>>>I had two commits to quickly verify the CI reliability:
>>>>
>>>>https://review.openstack.org/#/c/114393/
>>>>
>>>>https://review.openstack.org/#/c/40296/
>>>>
>>>>
>>>>I would expect all plugins and drivers passing on the first one and 
>>>>failing for the second but I got so many surprises.
>>>>
>>>>Neutron code quality and reliability is a top priority, if you 
>>>>ignore this report that plugin/driver will be candidate to be remove 
>>>>from Neutron tree.
>>>>
>>>>Cheers,
>>>>
>>>>Edgar
>>>>
>>>>P.s. I hate to be the inquisitor here҆ but someone has to do the 
>>>>dirty job!
>>>>
>>>>
>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mest...@mestery.com> wrote:
>>>>
>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests.
>>>>>Per the requirements wiki page here [1], everyone needs to be 
>>>>>running more than just Tempest API tests, which I still see most 
>>>>>neutron third-party CI setups doing. I'd like to ask everyone who 
>>>>>operates a third-party CI account for Neutron to please look at the 
>>>>>link below and make sure you are running appropriate tests. If you 
>>>>>have questions, the weekly third-party meeting [2] is a great place 
>>>>>to ask questions.
>>>>>
>>>>>Thanks,
>>>>>Kyle
>>>>>
>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
>>>>>
>>>>>_______________________________________________
>>>>>OpenStack-dev mailing list
>>>>>OpenStack-dev@lists.openstack.org
>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>_______________________________________________
>>>>OpenStack-dev mailing list
>>>>OpenStack-dev@lists.openstack.org
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>_______________________________________________
>>>>OpenStack-dev mailing list
>>>>OpenStack-dev@lists.openstack.org
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to