07.01.2013, 02:23, "Juan Jose Garcia-Ripoll" <juanjose.garciarip...@gmail.com>: > Dear Anton, thanks again for your work on this and for the good news.. It > seems your manual comparison of versions works better than the automated > process I tried to construct. Would it be very difficult to set up your > process in a more robust manner than mine? > > Best, > > Juanjo
Hello Juanjo. Sorry it took long to reply. I want to say that the approach is not so manual. I run few commands (git pull, ecl-rebuild.sh, run-agent.sh) and computer tests 800+ libraries on the new ECL. Then I run another command from slime and have a comparison of these results with results of the old version. I would say it's an automated system, but the user interface consists of a lisp script opened in emacs, where I run test-grid-reporting::print-compiler-diff on desired versions (or print all results, or print load failures sorted by number of blocked dependencies) I think this level of automation is the most convenient. We may add some more automation, for example you already have a setup where new commits initiate a test run. But IMHO it's not the biggest priority, because an automated system will require coding, debugging, modification and maintenance by itself, and will disctruct you from doing these activities on ECL. And will add relatevely little convenience, because starting run-agent.sh manually is easy. What is not so easy, but really deserves time and attention is reviewing the difference between two versions, analyzing whether the regressions are caused by ECL or it is a problem in environment/infrastructure. Then fixing ECL it it was a bug. This can not be automated. Example of what I mean by infrustructure problem is https://github.com/quicklisp/quicklisp-client/issues/71. It lays in inter-operation between Quicklisp, CCL and ECL: if during quicklisp:update-all-dists performed by CCL some ASDF systems with dot in file name were updated, then ECL can not load these systems, because CCL escaped dots in pathname when storing the pathname in quicklisp system file. I encountered this when testing ECL recently, and it took some time to investigate. Unfortunately, this kind of problems is unavoidable, they will happen sometimes. To minimize your efforts I believe it is not necessary to test every commit, it's enough to run tests once in a several weeks or in a month. So, after the above general thoughts, the concrete actions I propose to do: 1. The key point I suggest to change is to not compare each version with it's predecessor, but instead compare HEAD with the last version reviewed and verified to not have regressions (what I call "base version" in previous emails). I propose you to start doing it manually. If you do this kind of testing, you will end up having 3 ECL binaries on your machine: HEAD, base version and previous release. Because when new Quicklisp is out, it's necessary to retest not only HEAD, but also the base version and previous release so that ECL test results are compared on the same Quicklisp. 4. Submit test results to online storage instead of local-result-store (i.e. using test-grid-storage module with storage name "ecl") This will make the results available for me too. Please send me your run-agent.lisp file, I will show how to change it. 3. Could you share the scripts which build this page: http://ecls.sourceforge.net/reports-generated/ecl/index.html I would like to see how you determine order between ECL version - just by date or somehow else. If we can determine HEAD automatically, then probably it's not difficult to include HEAD/base-version comparison into the automated process. Best regards, - Anton PS Below is the script I currently have in slime session when analyze ECL test results. I just change the version numbers, execute the functions calls I want and review HTML: (ql:quickload :test-grid-storage) (ql:quickload :test-grid-reporting) (defpackage #:ecl-reports (:use :cl) (:import-from #:test-grid-reporting #:lisp #:libname #:subset #:lib-world #:results-cell-printer)) (in-package #:ecl-reports) (defparameter *db-snapshot-file* (asdf:system-relative-pathname :test-grid-storage "db-ecl.lisp")) (defparameter *r* (test-grid-storage:make-replica "ecl" *db-snapshot-file*)) (test-grid-storage:sync *r*) (defparameter *all-results* (test-grid-reporting::list-results (test-grid-storage:data *r*))) (test-grid-reporting::print-compiler-diff "ecl/ecl-diff-bytecode.html" *all-results* "quicklisp 2012-12-23" "ecl-12.12.1-ebafa5f2-linux-x86-bytecode" "ecl-12.12.1-89a8201b-linux-x86-bytecode") (test-grid-reporting::print-compiler-diff "ecl/ecl-diff-lisp-to-c.html" *all-results* "quicklisp 2012-12-23" "ecl-12.12.1-b6519e5c-linux-x86-lisp-to-c" "ecl-12.12.1-89a8201b-linux-x86-lisp-to-c") (test-grid-reporting::print-pivot "ecl/ecl-results.html" (subset *all-results* (lambda (result) (and (string= "quicklisp 2012-12-23" (lib-world result)) (some (alexandria:rcurry #'search (lisp result)) (list "12.12.1-unknown" "12.12.1-ebafa5f2" "12.12.1-89a8201b"))))) :rows '((libname string<)) :cols '((lib-world string>) (lisp string<)) :cell-printer #'results-cell-printer) ------------------------------------------------------------------------------ Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS, MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft MVPs and experts. SALE $99.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122412 _______________________________________________ Ecls-list mailing list Ecls-list@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ecls-list
