Thanks for the comments, Eli. I will act on most of them, I have a few questions below though.
On Wed, May 27, 2009 at 11:27 PM, Eli Barzilay <e...@barzilay.org> wrote: > On May 27, Jay McCarthy wrote: >> I'd like to get some comments on what I have so far and stuff I plan >> on doing. > > In complete as-I-see-it random order: > > * Nice interface, but the links are kind of easy to miss (since some > are the usual blue, and some are black). > > * I like the "alert only on change" policy -- that solves a bunch of > issues. But it might be problematic if there are files that produce > random output -- so plan on having the meta-data should hold > information like directions to ignore the file, the output, stderr > output, exit code, etc. > > * If you're already computing the durations, then this would be > another good point: if the time is very different than the previous > measurement, then it's worth an alert. Also, it might be good to > keep some historical data (perhaps on some selected files) to see > how mzscheme is doing in general. (The plan is that I'll eventually > add that to the nightly build too.) > > * Also, allowing to sort the result list on the time could be a useful > indicator that something is wrong. > > * You can probably tackle timeouts in a similar way. Another thing to > do with things like drscheme and plt-games is have the meta-data > specify that a new window is expected to pop up with some specific > title, and just declare success when you see it (see > "iplt/build/test-drscheme.ss" -- it does that for drscheme). > > * It might be better to read a file and run it only if it's a module > by the expected name, to avoid running bogus files. (Looks like > this is the bulk of the loading errors.) Since this depends only on > the contents of the file, you could just cache it. (Perhaps by just > catching the right load error and ignoring it.) > > * Links from a file's page to the svn location would be nice. Maybe > even dump the file's contents on the same page. (Makes it much > easier to see that some problem is expected.) > > * The font used for the file rows is different from the Total row. > (Makes it weird when there are two 1s that look different...) > > * Also, the meta-data should contain instructions that apply to a > whole collection -- this will be usful in the case of > tests/mzscheme, where it doesn't make sense to load each file, since > there's a central file that loads them all anyway. (So it should > probably look like some inclusion exclusion thing, where I can say > "run only all.ss like that, and ignore all other files".) > > * If you're running a separate process for each file, you can just as > well clean the directory and avoid those problems. Which directory do you mean here? > * I'll repeat a point I made earlier -- putting the meta data on svn > properties means that some people will never use that. I'd prefer > some file (maybe in iplt/build, maybe even using the same language > that is used for the specs). I don't understand this point. If it is some file in iplt then there's an even bigger disincentive for people to update it themselves. If it is a property it is not terribly hard to do a: svn pset plt:tester:command-line "mzscheme -qu ~a" the-file.ss or svn pset plt:tester:timeout "60" the-file.ss Maybe you are saying that in your experience people won't ever do anything, so I might as well make it easier for me. > >> Combining this with the saving of terminal output is, IMHO, a fairly >> robust way to locate errors and be testing suite agnostic. Basically, >> there will be another column that says "Changed?" and whether the >> output has changed. This is will be the basis of the "nag" emails with >> some heuristics to avoid unnecessary naggery. > > But when would "change" go away? My best guess for this involves > people logging in and clicking somewhere to acknowledge that a change > is expected. These changes would be computed on a per-revision basis. The only data is whether it is changed relative to the last revision. Therefore, there's no reason to acknowledge anything on the part of users. Am I misunderstanding your question? > >> For example, if the file you just edited in the commit displays >> something different, it won't consider it any error if it always >> displayed something in that way. For example, if X.ss never printed >> anything on stderr than it will nag if it starts to even though you >> just edited it. However, if X.ss always printed on stderr, it won't >> nag if it prints something different. Obviously these heuristics will >> be very fluid. > > I think that nagging also on a different output would be good, with a > meta-data way to avoid it. > >> * Using Subversion properties to set the timeout on a per-file basis >> >> This will help the build not wait for ever for DrScheme, as it won't >> complete. By including it in Subversion, it will be versioned with >> the file, so the metadata is not in a magical place on the server. > > The expected window title works better for this, IMO. Adds a test > instead of a fragile timeout. (And BTW, timeouts can be very fragile, > since the OS might do something like a backup which will throw > everything off.) This machine will do nothing else (including backups, because I could just rerun the software... everything it does is soft state.) > >> * Emitting the status of a build on twitter > > (Is that *really* necessary...?) Well, I use twitter, so it would be nice for me to know what the server is doing without getting lots of emails or having to check a site. (My main point in including the bullet is that you won't shouldn't expect informative emails unless something bad happened.) > >> * Eventually I'd like to do two different kinds of builds: A "fast" >> build that uses the previous slow build, but updates to the next >> version and perhaps does some sort of dependency heuristic to not >> run everything. A "clean" build corresponding to this. The goal >> would be that the fast build would be done within 30 minutes, but >> the clean would be available in a few hours. > > The time to run each of the files is probably far more dominating than > the time to do a full build. We're using the word "build" differently. I mean: everything my software does, i.e., running all the files. My "fast" version would not run _everything_ but would have some heuristic about what files should be run given commits. (Such as, everything in the same collection.) > >> It took 7.34 hours to do the build on my Macbook in power save mode. >> At the average of 8 commits per day, this is too slow to keep up >> with the edge, but perhaps a better machine will do better. I >> currently don't even run test two files at once. (I could because I >> have dual CPUs on the laptop.) Plus with better timeouts I could >> shave off almost 5 hours. > > IME, getting a linux installation on a good recent machine will be > much faster. > >> It took 500MBs of space for the source & compiled source. > > Actually I have that at 600MB now. (Which can lead to some issues > when dealing with the nightly builds -- since that's multiplied by the > number of builds, then there are several packages for each OS...) > > -- > ((lambda (x) (x x)) (lambda (x) (x x))) Eli Barzilay: > http://www.barzilay.org/ Maze is Life! > -- Jay McCarthy <j...@cs.byu.edu> Assistant Professor / Brigham Young University http://teammccarthy.org/jay "The glory of God is Intelligence" - D&C 93 _________________________________________________ For list-related administrative tasks: http://list.cs.brown.edu/mailman/listinfo/plt-dev
