On 21-08-2024 07:13, Randy MacLeod wrote:
On 2024-08-16 11:54 a.m., [email protected] wrote:
From: Yash Shinde<[email protected]>
Add some of the upsupported/failing tests to the exclude list
and ignore the failing unit tests.
Signed-off-by: Yash Shinde<[email protected]>
---
meta/lib/oeqa/selftest/cases/rust.py | 34 +++++++++-
.../rust/files/rust-oe-selftest.patch | 64 ++++++++++++++++++-
2 files changed, 95 insertions(+), 3 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/rust.py
b/meta/lib/oeqa/selftest/cases/rust.py
index 62ca762cd0..aa5814eccf 100644
--- a/meta/lib/oeqa/selftest/cases/rust.py
+++ b/meta/lib/oeqa/selftest/cases/rust.py
@@ -205,7 +205,39 @@ class RustSelfTestSystemEmulated(OESelftestTestCase,
OEPTestResultTestCase):
'library/std/src/sync/rwlock/tests.rs',
'library/std/src/thread/tests.rs',
'library/test/src/stats/tests.rs',
- 'tidyselftest'
+ 'tidyselftest',
+ 'compiler/rustc_span',
+ 'compiler/rustc_expand',
+ 'tests/ui/array-slice-vec',
+ 'tests/ui/associated-consts',
+ 'tests/ui/associated-types',
+ 'tests/ui/allocator',
+ 'tests/ui/abi/extern',
+ 'tests/codegen/non-terminate/infinite-loop-1.rs',
+ 'tests/codegen/coercions.rs',
+ 'tests/codegen/const_scalar_pair.rs',
+ 'tests/codegen/cold-call-declare-and-call.rs',
+ 'tests/codegen/issues',
+ 'tests/codegen/bool-cmp.rs',
+ 'tests/codegen/ascii-char.rs',
+ 'tests/codegen/frame-pointer.rs',
+ 'tests/codegen/common_prim_int_ptr.rs',
+ 'tests/codegen/default-requires-uwtable.rs',
+ 'tests/codegen/debug-accessibility/public-enum.rs',
+ 'tests/codegen/meta-filecheck/filecheck-flags.rs',
+ 'tests/codegen/float_math.rs',
+
'tests/codegen/debug-accessibility/tuple-fields.rs',
+ 'tests/codegen/consts.rs',
+ 'tests/codegen/enum/enum-bounds-check.rs',
+ 'tests/codegen/debug-alignment.rs',
+ 'tests/codegen/export-no-mangle.rs',
+ 'tests/codegen/fastcall-inreg.rs',
+
'tests/codegen/debuginfo-inline-callsite-location.rs',
+ 'tests/codegen/enum/enum-debug-tagged.rs',
+ 'tests/codegen/adjustments.rs',
+
'tests/codegen/macos/x86_64-macosx-deployment-target.rs',
+ 'tests/codegen/frame-pointer.rs',
+ 'tests/codegen/meta-filecheck/check-prefix.rs'
]
Hi Yash,
There was a problem with a SDK test failing in 1.78 due to a linkage
problem for rustc in the SDK.
You replied to the email saying:
"This also resolves the "bitbake core-image-sato -c do_testsdk"
error for cargo."
but I don't see a patch or change that would resolve that problem. Can
you explain briefly what
the problem was and what you did to fix it for me and anyone who might
be interested, please?
The problem was related to the changes in cargo with the following patch:
repro-issue-fix-with-cc-crate-hashmap.patch
The patch was implemented to fix the rust reproducibility issue.
I had successfully implemented and tested the patch to upgrade rust to
v1.78 version.
However, the issue was observed after integrating the rust repro issue fix.
The issue seemed to be somehow related to the checksum and json related
files.
After encountering the cargo SDK test error, I spent quite a lot of time
to debug the error from the logs.
It was observed that the issue was related to the cargo changes in
"vendor/cc-1.0.79/.cargo-checksum.json" file from repro issue fix patch
file.
Hence, I implemented the version upgrade to v1.78 from scratch and fixed
the issue.
It seems that we now have just over 170 tests that we're skipping! Ugh!
Skipping tests is okay as a temporary work-around but once we have
reached feature freeze next week on Aug 26,
please start looking at why these tests are failing and try to fix the
root cause so the tests can be re-enable.
$ rg -A 175 "exclude_list = " meta/lib/oeqa/selftest/cases/rust.py
Where should you start, you might ask? Let's look at this list...
Oh, it's not sorted alphabetically as we agreed it should be!
Please fix that as a single commit now or as part of the next Rust
update if that makes more sense for you
given that the feature freeze deadline is less than a week away.
BTW, you should be able to do this sorting within your editor (emacs
can do it at least! ;-) ) but let's do it in the shell:
# cut on the ' delimiter:
$ rg -A 172 "exclude_list = " meta/lib/oeqa/selftest/cases/rust.py |
tail -172 | cut -d"'" -f2 | sort > /tmp/after
The 172 was determined manually.
Here's why you want the list sorted:
❯ uniq -c /tmp/after | grep -v " 1 "
2 compiler/rustc_interface/src/tests.rs
2 library/std/src/thread/tests.rs
2 library/test/src/stats/tests.rs
2 tests/codegen/frame-pointer.rs
2 tests/rustdoc-json
2 tests/rustdoc-ui/doc-test-rustdoc-feature.rs
2 tests/rustdoc-ui/no-run-flag.rs
so there are "only" 165 tests being skipped:
❯ uniq -c /tmp/after | wc -l
165
I'm most concerned about the compiler and library/std tests but
we do need to work on understanding and hopefully fixing all the
skipped tests.
Please take a look at say up to 5 tests, hopefully related, and work
on fixing them as a group,
rather than trying to fix the entire list in one work item and one commit.
We have already started to work on clearing the exclude list based on
rust v1.77.2.
We have removed around 50 tests from the exclude list and are being
successfully tested(including a few from compiler and library/std tests).
Currently, the testing is being performed with v1.78 and will be
submitted upstream if there are no issues.
There are plans to improve this process further for v1.80 by removing
more tests from the exclude list.
In future commits, please include a summary of the total number of
tests that are being skipped
and the total number that are being run so we can have a numerical
indicator to track the rust
quality.
Oh, you had exact statistics in this poky commit:
commit c496b1eaf4b72ca979d47c37e73e44cd145fed1c
Author: Yash Shinde <[email protected]>
Date: Sat Jul 29 03:00:21 2023
...
* The tests that are passed and skipped are as follows-
Target PASS SKIPPED
ARM 15507 428
ARM64 15535 400
MIPS64 15479 456
X86 15528 407
X86-64 15643 292
so it looks like things have improved. Yay!
Do you know offhand if upstream Rust passes all tests for x86-64 and
arm64 ?
I know that running these tests takes quite a long time so perhaps you
can run
the full testsuite for all targets once per YP release, on the final
Rust update so
that the results are in a commit log. Given that I've changed
requirements, late in this release,
if that's a problem then just send the update to 1.80.x in without the
full set of all target statistics.
Hopefully during the M4 bug fix cycle, you can fix some of the failing
tests and provide statistics
for all targets, including riscv64.
Please let me know what you think.
Thanks Randy. This seems to be a great idea to measure the overall rust
quality.
We will provide the summary of test results for set of all target
statistics along with the upgrades.
Mips, PPC and Riscv are tier-2 and tier-3 targets for which even rust
upstream doesn't do
official tests. So, they may or may not work.
Many of the tests in the exclude lists come from mips and ppc failures.
We are investigating more about the test failures that occurs only for
tier-2 and tier-3 targets.
We will prepare a report on the same and share it with you after the
analysis.
Please refer tier 2 and tier 3 section from the following rust document:
https://doc.rust-lang.org/nightly/rustc/platform-support.html
Regards,
Yash
Thanks,
../Randy
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#203583):
https://lists.openembedded.org/g/openembedded-core/message/203583
Mute This Topic: https://lists.openembedded.org/mt/107934730/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
