On Tue, Mar 2, 2010 at 5:57 PM, David A. Ramos <daramos at stanford.edu> wrote: > Hi Cristi, > >> I was wondering if there is a way to use run.istats to generate the >> coverage just for the interesting source files (i.e., not for the >> library files). If this is possible, it wouldn't be necessary to >> replay the test cases and use gcov to get the coverage results.
Yes, its certainly possible to use the istats to get more information. However, from a purity perspective there is something nice about getting the data from replayed runs with gcov, since that prevents bugs in KLEE from skewing any results. My zcov tool could easily be adapted to support loading information that KLEE spit out directly (http://minormatter.com/zcov). > > We actually have a feature in our local Stanford KLEE repository that accepts > a file with a list of routines to ignore for the purposes of calculating > coverage statistics. I don't think it affects the run.istats output, though. > It definitely affects the output of "--write-cov," which outputs files > listing the newly covered lines for each testcase generated. I'll submit a > patch to Daniel to add this functionality to the public repo (and CC you). I actually removed this feature intentionally from the open source version because I wasn't very happy with the design and implementation. If people find it useful, I'm not opposed to adding it back although I would prefer to solve the problem differently. >> In your experience, does run.istats reliably show the coverage >> information (or the uncovered lines) when running with klee >> --optimize? I tried with echo.bc and did not see what I was expecting, >> kcachegrind did not display any uncovered lines in echo.c, as it was >> supposed to. > > Some of the LLVM optimization passes tend to obscure the original source > lines, unfortunately. I've been noticing this, too. For truly representative > coverage on optimized programs, gcov is really the only way to go. Any other > method is just an approximation. Daniel, please correct me if I'm wrong here. LLVM optimization *used* to completely obliterate debug info. Nowadays there is ongoing work to make the optimizations preserve line information, but that is still very much a work in progress. Any application which wants some kind of reliable information about the coverage probably needs to take an instrumentation approach a la gcov. Eventually, I hope, someone will implement gcov like features in LLVM/Clang and we could use that as a basis for a better system. >> Finally, since Klee generates statistics when it covers new lines, >> would it be possible to take the information from run.istats and view >> what coverage results were obtained at each moment of time? (Something >> like an iterative coverage monitor). > > I don't think this can be done at the moment. You can graph the total > coverage percentages over time using run.stats. This won't give you the > individual source lines, though. I personally use the run stats information as a runtime monitor, in conjunction with some emacs foo which just runs a command periodically in a separate frame. I just leave it there running klee-stats on a directory which is useful for watching the summary. If you want more detailed information, it should be easy enough to collect it. A simple approach would be to just check point the istats files every once in a while. A more sophisticated approach could use the TreeStreamer ADT to monitor all the coverage data, even partitioned by state. I have used the TreeStreamer path information to generate some movies of the state space exploration over time. For example: http://minormatter.com/movie_1.avi The nice thing about the TreeStreamer approach is it is relatively efficient at capturing boatloads of information, if you really want to do a comprehensive analysis later. - Daniel > -David > > _______________________________________________ > klee-dev mailing list > klee-dev at keeda.stanford.edu > http://keeda.Stanford.EDU/mailman/listinfo/klee-dev >
