Make changes to the patchtest script, helper modules, test suites, and selftest
files to address various shortcomings with the current version, including:
- Re-enable the previously included mergeability tests, which have been broken
since prior to the 2023 Yocto RFQ
- Improve series testing by fixing signal handling and return code logic for the
'--directory' flag, and by making it possible for patchtest to report the
shortlog for failing commits when testing series as single files
- Make branch checking for merge tests more robust but retain flexibility for
the selftest suite, and so that users can test series against branches other
than the target one
- Add new tests for malformed lines and whitespace in the TestPatch module
- Refactors across the entire suite to remove dead code, simplify logic, and
update selftest files
- Expand the selftest file suite for all of the new cases added, bringing us
from 38 to 51 selftests.
Before:
|============================================================================
|Testsuite summary for patchtest
|============================================================================
|# TOTAL: 38
|# XPASS: 18
|# XFAIL: 18
|# XSKIP: 2
|# PASS: 0
|# FAIL: 0
|# SKIP: 0
|# ERROR: 0
|============================================================================
After:
|============================================================================
|Testsuite summary for patchtest
|============================================================================
|# TOTAL: 51
|# XPASS: 25
|# XFAIL: 26
|# XSKIP: 0
|# PASS: 0
|# FAIL: 0
|# SKIP: 0
|# ERROR: 0
|============================================================================
As always, there's lots of room for patchtest to grow further (I'd still very
much like to convert it to a standalone Python tool one day). However, this is
already a sizeable rework, and it should bring immediate benefit to anyone using
it. I've made a point to label each individual patch appropriately, but for
emphasis this work has been done in tandem with the Claude AI, specifically
Sonnet 4.6 and 4.7.
Given the skepticism that many of us have towards AI tooling, I made a point to
do a review of the changes Claude had made myself before performing the final
squash and rebase. This did reveal some bloat that I've since removed, but for
the most part it made functional and useful changes. The selftest suite was
routinely triggered locally to ensure it didn't cause any bizarre regressions,
and new selftest files were added immediately for new test cases (or, in the
case of the mergeability tests, refactored so that they were no longer labeled
as skips). I've also made a point to pull patch series at random from the
mailing list and test directly.
Parts of this series (such as the '--directory' usage improvements) were written
to address the needs of folks at Smile. Others address one or more outstanding
bugs in Bugzilla, which I haven't consistently labeled as such (but I'll close
them when this gets merged). The next priority will be to improve the actual
deployment of the Patchtest service on the Yocto Project infrastructure.
Trevor Gamblin (11):
patchtest: check for meta-selftest, cleanup script
patchtest: test series mergeability
patchtest: tests: cleanup test suites
patchtest: correctly abort --directory test
patchtest: fix branch applicability reporting
patchtest: test_patch: improve patch-specific tests
patchtest: test_mbox: skip cover letters
patchtest: mbox.py: improve 'From' line matching
patchtest: selftest: stop if patchtest fails
patchtest: selftest: rework test files
patchtest: selftest: extend test file suite
meta/lib/patchtest/mbox.py | 56 ++---
meta/lib/patchtest/patchtest_parser.py | 2 +-
meta/lib/patchtest/patchtest_patterns.py | 5 +
meta/lib/patchtest/repo.py | 48 +++++
...x.pretest_series_applies_on_branch.1.pass} | 22 +-
...x.pretest_series_applies_on_branch.2.fail} | 0
.../TestMbox.test_bugzilla_entry_format.fail | 5 +-
.../TestMbox.test_bugzilla_entry_format.pass | 5 +-
...TestMbox.test_commit_message_presence.fail | 5 +-
...TestMbox.test_commit_message_presence.pass | 5 +-
...tMbox.test_commit_message_user_tags.1.pass | 5 +-
...tMbox.test_commit_message_user_tags.2.pass | 5 +-
...estMbox.test_commit_message_user_tags.fail | 5 +-
.../files/TestMbox.test_mbox_format.pass | 5 +-
.../files/TestMbox.test_non_auh_upgrade.fail | 25 +++
.../files/TestMbox.test_non_auh_upgrade.pass | 25 +++
.../files/TestMbox.test_shortlog_format.fail | 5 +-
.../files/TestMbox.test_shortlog_format.pass | 5 +-
.../files/TestMbox.test_shortlog_length.fail | 5 +-
.../files/TestMbox.test_shortlog_length.pass | 5 +-
...estMbox.test_signed_off_by_presence.1.fail | 5 +-
...estMbox.test_signed_off_by_presence.2.fail | 5 +-
.../TestMbox.test_signed_off_by_presence.pass | 5 +-
.../TestMbox.test_target_mailing_list.1.fail | 26 +++
.../TestMbox.test_target_mailing_list.2.fail | 24 +++
.../TestMbox.test_target_mailing_list.pass | 25 +++
.../TestMetadata.test_max_line_length.fail | 26 +++
.../TestMetadata.test_max_line_length.pass | 25 +++
.../files/TestPatch.test_cve_tag_format.fail | 5 +-
.../files/TestPatch.test_cve_tag_format.pass | 5 +-
...estPatch.test_patch_malformed_context.fail | 38 ++++
...estPatch.test_patch_malformed_context.pass | 38 ++++
...tPatch.test_patch_trailing_whitespace.fail | 38 ++++
...tPatch.test_patch_trailing_whitespace.pass | 38 ++++
...TestPatch.test_signed_off_by_presence.fail | 5 +-
...TestPatch.test_signed_off_by_presence.pass | 5 +-
....test_upstream_status_presence_format.fail | 61 ++++++
....test_upstream_status_presence_format.pass | 62 ++++++
meta/lib/patchtest/selftest/selftest | 25 ++-
meta/lib/patchtest/tests/base.py | 117 ++++------
meta/lib/patchtest/tests/test_mbox.py | 56 +++--
meta/lib/patchtest/tests/test_metadata.py | 112 +++++-----
meta/lib/patchtest/tests/test_patch.py | 113 ++++++++--
.../lib/patchtest/tests/test_python_pylint.py | 8 +-
scripts/patchtest | 200 +++++++++---------
45 files changed, 904 insertions(+), 406 deletions(-)
rename
meta/lib/patchtest/selftest/files/{TestMbox.test_series_merge_on_head.1.skip =>
TestMbox.pretest_series_applies_on_branch.1.pass} (66%)
rename
meta/lib/patchtest/selftest/files/{TestMbox.test_series_merge_on_head.2.skip =>
TestMbox.pretest_series_applies_on_branch.2.fail} (100%)
create mode 100644
meta/lib/patchtest/selftest/files/TestMbox.test_non_auh_upgrade.fail
create mode 100644
meta/lib/patchtest/selftest/files/TestMbox.test_non_auh_upgrade.pass
create mode 100644
meta/lib/patchtest/selftest/files/TestMbox.test_target_mailing_list.1.fail
create mode 100644
meta/lib/patchtest/selftest/files/TestMbox.test_target_mailing_list.2.fail
create mode 100644
meta/lib/patchtest/selftest/files/TestMbox.test_target_mailing_list.pass
create mode 100644
meta/lib/patchtest/selftest/files/TestMetadata.test_max_line_length.fail
create mode 100644
meta/lib/patchtest/selftest/files/TestMetadata.test_max_line_length.pass
create mode 100644
meta/lib/patchtest/selftest/files/TestPatch.test_patch_malformed_context.fail
create mode 100644
meta/lib/patchtest/selftest/files/TestPatch.test_patch_malformed_context.pass
create mode 100644
meta/lib/patchtest/selftest/files/TestPatch.test_patch_trailing_whitespace.fail
create mode 100644
meta/lib/patchtest/selftest/files/TestPatch.test_patch_trailing_whitespace.pass
create mode 100644
meta/lib/patchtest/selftest/files/TestPatch.test_upstream_status_presence_format.fail
create mode 100644
meta/lib/patchtest/selftest/files/TestPatch.test_upstream_status_presence_format.pass
--
2.54.0
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#237047):
https://lists.openembedded.org/g/openembedded-core/message/237047
Mute This Topic: https://lists.openembedded.org/mt/119319575/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
