[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 David Malcolm changed: What|Removed |Added Status|ASSIGNED|RESOLVED Resolution|--- |MOVED --- Comment #7 from David Malcolm --- Marking this one as RESOLVED MOVED, since I'm now doing this in: https://github.com/davidmalcolm/gcc-analyzer-integration-tests
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 --- Comment #6 from David Malcolm --- I've created https://github.com/davidmalcolm/gcc-analyzer-integration-tests which builds Juliet plus various real-world C projects with a candidate build of GCC with -fanalyzer and captures the diagnostics in SARIF format. https://github.com/microsoft/sarif-tools has a command for comparing SARIF files (or pairs of directories full of SARIF files) which is what I'm currently using to compare diagnostics emitted before/after a candidate GCC patch. I'm working through the diagnostics seen with GCC trunk's -fanalyzer, classifying them into true vs false positives and will probably write my own comparison script to show the effect of a GCC patch on false vs true positives.
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 --- Comment #5 from David Malcolm --- Another source of possible benchmarks: https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks >From SV-COMP: https://sv-comp.sosy-lab.org/ This embeds the Juliet testsuite, but also many other tests.
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 --- Comment #4 from CVS Commits --- The master branch has been updated by David Malcolm : https://gcc.gnu.org/g:860c56b5bc356960a4d0445dadc43ceddbe3c7e2 commit r12-5701-g860c56b5bc356960a4d0445dadc43ceddbe3c7e2 Author: David Malcolm Date: Wed Dec 1 14:12:33 2021 -0500 analyzer: fix false leak seen in Juliet 1.3 [PR102471] Juliet 1.3's CWE415_Double_Free__malloc_free_*_67a.c were showing leak false positives in non-LTO builds; fixed thusly. gcc/analyzer/ChangeLog: PR analyzer/102471 * region-model-reachability.cc (reachable_regions::handle_parm): Treat all svalues within a compound parm has reachable, and those wrapped in a cast. gcc/testsuite/ChangeLog: PR analyzer/102471 * gcc.dg/analyzer/leak-3.c: New test. Signed-off-by: David Malcolm
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 David Malcolm changed: What|Removed |Added Status|NEW |ASSIGNED --- Comment #3 from David Malcolm --- I've got a primitive workflow of running the Juliet 1.3 testsuite, and a script for testing results here: https://github.com/davidmalcolm/juliet-harness Currently I've only run it on the: C/testcases/CWE415_Double_Free/s01 sudirectory, with: # of passes 130 # of failures 103 I'm looking through the failures.
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471
--- Comment #2 from David Malcolm ---
There's also
https://www.nist.gov/itl/ssd/software-quality-group/other-assurance-tool-test-collections
Currently the only non-Java collection on the list is:
https://sir.csc.ncsu.edu/portal/index.php
However that repository has:
https://sir.csc.ncsu.edu/portal/sir-license.php
which seems to be a non-Open Source licence ("solely for non-commercial,
educational, evaluation and/or personal use").
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 Andrew Pinski changed: What|Removed |Added Severity|normal |enhancement
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 Richard Biener changed: What|Removed |Added Status|UNCONFIRMED |NEW Last reconfirmed||2021-09-24 Ever confirmed|0 |1
[Bug analyzer/102471] RFE: add support to analyzer testsuite for running SAMATE/SARD tests (e.g. Juliet Test Suite)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 --- Comment #1 from rguenther at suse dot de --- On Thu, 23 Sep 2021, dmalcolm at gcc dot gnu.org wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102471 > > Bug ID: 102471 >Summary: RFE: add support to analyzer testsuite for running > SAMATE/SARD tests (e.g. Juliet Test Suite) >Product: gcc >Version: 12.0 > Status: UNCONFIRMED > Severity: normal > Priority: P3 > Component: analyzer > Assignee: dmalcolm at gcc dot gnu.org > Reporter: dmalcolm at gcc dot gnu.org > CC: rguenth at gcc dot gnu.org > Target Milestone: --- > > See: > https://www.nist.gov/itl/ssd/software-quality-group/samate > https://samate.nist.gov/SARD/testsuite.php > > The links above have various promising-looking testsuites e.g. > - Juliet Test Suite > - Klocwork test suite > - ITC-Benchmarks > etc > > It would be good to be able to (somehow) automatically run them as part of > regression testing of the analyzer - either by turning them directly into > DejaGnu tests, or by wrapping the suite's own harness in a way that we can > invoke it during "make check". It might be also feasible to think of on-the-side CI runs of external testsuites on buildbots or similar and have results reported on gcc.gnu.org.
