Michael Droettboom wrote: > Interesting result. I pulled all of the new "actual" files from the 21 > failing tests on the buildbots to my local machine and all of those > tests now pass for me. Good. Interestingly, there are still two tests > failing on my machine which did not fail on the buildbots, so I can't > grab the buildbots' new output. Well, if they're not failing on the buildbots, that means the baseline in svn can't be too different than what they generate. But it's a good point that we want the actual output of the buildbots regardless of whether the test failed. > Could this just be a thresholding issue > for the tolerance value? I'm a little wary of "polluting" the baseline > images with images from my machine which doesn't have our "standard" > version of Freetype, so I'll leave those out of SVN for now, but will go > ahead and commit the new baseline images from the buildbots. Looking at the 2 images failing on the buildbots, I'm reasonably sure they were generated by James Evans when he created the first test infrastructure. So I say go ahead an check in the actual images generated by the buildbots. (Or did you recently re-upload those images?) > Assuming > these two mystery failures are resolved by pulling new images from the > buildbots, I think this experiment with turning of hinting is a success. > Yes, I think so, too. I was going to suggest getting on the freetype email list to ask them about their opinion on what we're doing. > As an aside, is there an easy way to update the baselines I'm missing? > At the moment, I'm copying each result file to the correct folder under > tests/baseline_images, but it takes me a while because I don't know the > heirarchy by heart and there are 22 failures. I was expecting to just > manually verify everything was ok and then "cp *.png" from my scratch > tests folder to baseline_images and let SVN take care of which files had > actually changed. Unfortunately, there's no easy baseline update yet. John wrote one for the old test infrastructure, but I ended up dropping that in the switchover to the simplified infrastructure. The reason was that the image comparison mechanism, and the directories to which they were saved, changed, and thus his script would have require a re-working. Given that I don't consider the current mechanism for this particularly good, I was hesitant to invest the effort to port over support for a crappy layout.
(The trouble with the current actual/baseline/diff result gathering mechanism is that it uses the filesystem as a means for communication withing the nose test running process in addition to communication with the buildbot process through hard-coded assumptions about paths and filenames. If the only concern was within nose, we could presumably re-work some the old MplNoseTester plugin to handle the new case, but given the buildbot consideration it gets more difficult to get these frameworks talking through supported API calls. Thus, although the hardcoded path and filename stuff is a hack, it will require some serious nose and buildbot learning to figure out how to do it the "right" way. So I'm all for sticking with the hack right now, and making a bit nicer by doing things like having a better directory hierarchy layout for the actual result images.) > This is just the naive feedback of a new set of eyes: > it's extremely useful and powerful what you've put together here. > Thanks for the feedback. The goal is that Joe Dev would think it's easy and useful and thus start using it. Tests should be simple to write and run so that we actually do that. Like I wrote earlier, by keeping the tests themselves simple and clean, I hope we can improve the testing infrastructure mostly independently of changes to the tests themselves. -Andrew ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
