On 30 Sep 2024, at 19:54, Aaron Conole wrote:
> Eelco Chaudron <[email protected]> writes:
>
>> On 28 Sep 2024, at 15:03, Aaron Conole wrote:
>>
>>> The 0-day robot has been testing patches for 6 years, and we've
>>> had support for other labs to integrate for 3. However, this
>>> isn't well documented, and has made it difficult for others to
>>> know how they can contribute to testing. To that end, this
>>> patch introduces some documentation for the 0-day robot and
>>> how to integrate into the patch reporting process.
>>
>> Thanks for adding this documentation, some small comments below.
>>
>> Cheers,
>>
>> Eelco
>>
>>> Signed-off-by: Aaron Conole <[email protected]>
>>> ---
>>> .../contributing/submitting-patches.rst | 3 +-
>>> Documentation/topics/testing.rst | 105 ++++++++++++++++++
>>> 2 files changed, 107 insertions(+), 1 deletion(-)
>>>
>>> diff --git
>>> a/Documentation/internals/contributing/submitting-patches.rst
>>> b/Documentation/internals/contributing/submitting-patches.rst
>>> index 8a8bc11b0a..c01ac7bbdc 100644
>>> --- a/Documentation/internals/contributing/submitting-patches.rst
>>> +++ b/Documentation/internals/contributing/submitting-patches.rst
>>> @@ -70,7 +70,8 @@ Testing is also important:
>>>
>>> If you are using GitHub, then you may utilize the GitHub Actions CI build
>>> systems. They will run some of the above tests automatically
>>> -when you push changes to your repository.
>>> +when you push changes to your repository. See the "Continuous Integration"
>>> +section in :doc:`/topics/testing` for details on continuous integration.
>>>
>>> Email Subject
>>> -------------
>>> diff --git a/Documentation/topics/testing.rst
>>> b/Documentation/topics/testing.rst
>>> index dcf10a4db2..49ee7d7ffa 100644
>>> --- a/Documentation/topics/testing.rst
>>> +++ b/Documentation/topics/testing.rst
>>> @@ -545,3 +545,108 @@ Once you are done with experimenting you can
>>> tear down setup with::
>>>
>>> Sometimes deployment of Proof of Concept may fail, if, for example, VMs
>>> don't have network reachability to the Internet.
>>> +
>>> +
>>> +Continuous Integration
>>> +----------------------
>>> +
>>> +The Open vSwitch project can make use of multiple public and hosted
>>> +CI services to help developers ensure their patches don't introduce
>>> +additional regressions. Each service requires different forms of
>>> +configuration, and for the supported services the configuration
>>> +file(s) to automatically build Open vSwitch with various build
>>> +configurations and run the testsuites is/are provided in the
>>
>> I think test suites are two words.
>
> I guess it could be either. From what I see in the documentation /
> comments, we use 'testsuite' more often than 'test suite'.
>
> I can switch it in v2.
Looks like we are not consistent in this document :) We use testsuite more than
test suite, so I can you can keep it like this.
>>> +repostiory. For example, the GitHub Actions builds will be performed
>>> +with gcc, clang, sparse, including DPDK, etc. with the -Werror
>>> +compiler flag included. Therefore, the build will fail if a new
>>> +warning gets introduced by a change.
>>> +
>>> +Each ci system uses a different method of enablement, but most are
>>> available
>>
>> CI with captials as in the rest of the text?
>
> Ack.
>
>>> +from the GitHub settings page. By default, Open vSwitch includes a GitHub
>>> +actions running configuration, and this will automatically email
>>> the repository
>>> +owner.
>>> +
>>> +Currently, Open vSwitch project enables the following public CI
>>> services along
>>> +with the appropriate configuration settings::
>>> +
>>> + - AppVeyor: appveyor.yml
>>> + - Cirrus-CI: .cirrus.yml
>>> + - GitHub Actions: .github/workflows
>>> +
>>> +GitHub Actions is available without any additional configuration.
>>> +
>>> +Additionally, as each patch is posted to the mailing list, the public CI
>>> +machinery will run additional tests on the patches and report results.
>>> +
>>> +Public report / Private lab hybrid testing
>>> +------------------------------------------
>>> +
>>> +The Open vSwitch project makes use of the ozlabs patchwork instance
>>
>> Maybe put in a link to the instance?
>
> Good idea.
>
>>> +to track patch status and management. This patch tracking tool
>>> +provides information to maintainers on the state of patches proposed
>>> +for Open vSwitch. The CI process for Open vSwitch makes use of the
>>> +checks feature of the ozlabs patchwork instance. These allow developers
>>
>> OzLabs
>>
>>> +and maintainers to see what tests have been run, and report pass / fail
>>> +criteria.
>>> +
>>> +In order to know that a patch or series is ready for testing, the
>>> +Open vSwitch project makes use of the "0-day Robot", which is a hosted
>>> +jenkins instance running the pw-ci_ scripts. These can monitor a
>>
>> Jenkins
>>
>>> +running patchwork instance for new patches and submit the patch details
>>> +to other build systems, like jenkins.
>>> +
>>> +Once a patch is tested, it would be good to report the results. To this
>>> +end, the Open vSwitch "0-day Robot" will accept emails sent to
>>> [email protected] formatted in the correct way to be reflected
>>> +on this page. This allows any lab to contribute to the testing and
>>> +validation of patches. Note that the ovs-build list participation
>>> +requires subscribing the reporting email account to the list.
>>> +
>>> +To report a test status to a particular patch, send exactly one email to
>>> +the mailing formatted as such::
>>> +
>>> + From: your email <[email protected]>
>>> + To: [email protected]
>>> + Date: Mon, 28 Jun 2021 00:00:00 +0000
>>> + Subject: |STATUS| pwPATCHID commit subject
>>> +
>>> + Test-Label: your-robot-or-test-name
>>> + Test-Status: STATUS
>>> + http://patchwork.ozlabs.org/api/patches/PATCHID/
>>> +
>>> + _ONE LINE DESCRIPTION_
>>> +
>>> + Addtional details
>>> +
>>> +In the above example, the STATUS is one of "success" "warning"
>>> "fail" depending
>>> +on the outcome of the test. This value is case-sensitive. PATCHID should
>>> be
>>> +the patch for which the test was executed, and ONE LINE DESCRIPTION
>>> should be a
>>> +simple one line result description, ex: "robot-test: success"
>>> +
>>> +It is strongly recommended that if the patch is reporting a failure case,
>>> one
>>> +of the patch authors is cc'd as well. Please try to keep the details of a
>>> +pass or fail to smallest amount that includes useful data. For example, if
>>> +a build log can be hosted in a publicly viewable way, then please include a
>>> +URL rather than the entire build log. If that isn't possible, try to parse
>>> +the test logs and include a small snippet with the relevent details.
>>> Always
>>
>> relevant details.
>>
>>> +also try to publish what the test is running - that can help developers to
>>> +recreate failures themselves.
>>> +
>>> +Example for success::
>>> +
>>> + From: 0-Day Robot <[email protected]>
>>> + To: [email protected]
>>> + Date: Fri, 25 Jun 2021 14:30:37 +0400
>>> + Subject: |success| pw1497375 [ovs-dev] [PATCH v2] checkpatch:
>>> Ignore macro definitions of FOR_EACH
>>> +
>>> + Test-Label: github-robot
>>> + Test-Status: success
>>> + http://patchwork.ozlabs.org/api/patches/1497375/
>>> +
>>> + _github build: passed_
>>> + Build URL: https://github.com/ovsrobot/ovs/actions/runs/972188336
>>> +
>>> +If possible, please also try to support the ``recheck-request`` feature
>>> that
>>> +the "0-day Robot" currently supports.
>>> +
>>> +.. _pw-ci: https://github.com/ovsrobot/pw-ci
>>
>>
>> Should we explain the recheck-request feature earlier in the documentation?
>
> I can move it earlier, makes sense to me.
Sorry, I was not clear. I wanted to say that earlier in the text, we should
explain how to use the recheck-request option in an email to the list to have a
failed test rerun.
>>
>>> --
>>> 2.46.1
>>>
>>> _______________________________________________
>>> dev mailing list
>>> [email protected]
>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
