23.07.2012, 17:30, "Juan Jose Garcia-Ripoll" <juanjose.garciarip...@gmail.com>: > Is it possible to reproduce your tests with my own platforms?
Yes, it is one of the primary goals to make it easy to run tests for everyone and to share the results. If you are interested, I will provide you instructions somewhat later, because I am now working on improvements of the test running program, and my work will affect the instructions, so it's better to discuss it after I finish the current tasks. As a short overview. I have a function libtest(libname). This function runs test suite of the specified library, downloading it with quicklisp if necessary, and converts the test result from the format of test framework of this particular library into a unified format. On top of this there is a program called test-grid-agent. It's a lisp program, which bootstraps private quicklisp in a local directory, runs each test suite in a separate lisp process, collects test statuses and logs and uploads them admin (to me) for publishing. I am now making some final changes in the agent and testing it. > Would it also be possible to get one table per platform? Customizing the reports would require some programming. Out of box only reports described on this overview page are ready: http://common-lisp.net/project/cl-test-grid/. The test result are stored in single lisp file [1] and based on the current reporting code [2] it is probably not difficult to get the presentation you need. [1] https://github.com/cl-test-grid/cl-test-grid/blob/master/db.lisp [2] https://github.com/cl-test-grid/cl-test-grid/blob/master/reporting/reporting.lisp In one of the next milestones I plan to improve the reports (simplify and document the test results data access, add filtering and matching functions). But I will switch to it after I will finish with the agent program. Could you explain what do you mean by one table per platform? And in general I am interested to know what a compiler maintainer would like to see in reports. > Many of the libraries seem to have failed tests also on other ports. Yes, there are lot of failures on other compilers. > Would it be possible to distinguish between test failure and build failure? > That would set up priorities straight. There is no separate test status for build failure (I saw on ECL test farm record build status and test status separately and considered but haven't decided to introduce it yet). There are cases when we can distinguish build failure in test-grid, but I don't want to overload you by the details now. Today you have 4 ECL version x 56 libraries = 224 test results. Lot for work to review everything. And we will add more libraries / more ECL versions. I came to a conclusion that a better strategy is not to review every test result, but to concentrate on regressions. We capture situation as it is, and then monitor how it changes. If yesterday libraryX reported 3 test cases failed, but today it has 4 test cases failed, then we know someone's recent changes cause this and we can react. I.e. a strategy of prevent the situation of moving into wrong direction. (BTW, the regression detection code would be able to detect if a library yesterday had test failures, but today it just doesn't build) Still, the short answer, you can't easily distinguish build failures in the test results. Best regards, - Anton ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Ecls-list mailing list Ecls-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ecls-list
