I've finished the core of the regression tester. You can now run it and generate a html report but you will need a few tools installed: imagick, xwd, npm, pretty-diff and xvfb. This should all be listed in ft-regression.sh's comments.If you want to run it locally make sure you have a ~/test-fonts with .ttf files in it. You will also need freetype2 and freetype2-demos sources next to each other. You can run it locally by calling: ./CI/ft-regression.sh <commit> eg: ./CI/ft-regression.sh HEAD~1 from inside your freetype2 directory. It should generate an easy to digest report to /tmp/ft-tests/index.html. This is basically what I wanted to accomplish for my GSOC project but I can do a lot of the things mentioned in my previous message if I pass my evaluation.
I've also got the testing apparatus running on azure pipelines with the other tests here: https://dev.azure.com/fundies/freetype2/_build/results?buildId=232&view=results Ignore the mingw build failures as they seem to be in the midst of a regime change and their new signed key databases aren't in sync with azure. You can download the report it generates from here: https://dev.azure.com/fundies/9eabb07a-6a4d-4b68-b22e-60f9e02c1927/_apis/build/builds/232/artifacts?artifactName=Archlinux%20Regression%20tests&api-version=5.1&%24format=zip The file is rather large. In the future, I could probably shrink it quite a bit by using 7zip instead of regular zip or I could split the tests into smaller chunks too. The report doesn't demonstrate the image comparison because there are obviously no regressions between my commit and master. However, if there is one it will generate a page where you can see the differences between images on mouse over. For text files it generates a diff report using pretty-diff. You can see this in the freetype-bench comparisons. As I've said this should be mostly working now. I also believe I've commented the source pretty thoroughly. I've also moved the scripts to a subdir as requested. If there are any changes / improvements you would like to see to the reports or scripts please let me know. If not, I have listed several things I would like to do to polish it in my previous message. Here is my patchset: https://patch-diff.githubusercontent.com/raw/fundies/freetype2/pull/1.patch On Tue, Jul 21, 2020 at 12:23 AM Werner LEMBERG <w...@gnu.org> wrote: > > Hello Greg, > > > > figured out the cause. I have however been working on this as much > > as time allows. I've mostly hashed out the scripts to run regression > > tests using demos here: > > > https://patch-diff.githubusercontent.com/raw/fundies/freetype2/pull/1.patch > > [...] > > Some comments. > > * There are too much top-level files. Please move them to > subdirectories. > > * Please submit patches to the list that are generic enough to be > included into 'master'. This affects the CMake stuff, for example. > > * If possible, don't use lines longer than 80 characters. > > * Please avoid trailing whitespace. > > > . The scripts still need a bit of work. I've only just started the > > bits that generate a html report for devs to inspect and I need to > > hook it into azure pipelines. > > My problem is that I have zero knowledge of azure. So please... > > > A couple other issues I want/plan to address are: > > > > 1. I need to comment the scripts better > > ... do that in the first place. > > > 2. I need to add a script argument to specify the demos commit as > > not all commits of freetype2 are compatible with freetype2-demos > > (in fact this broke on me over the past couple weeks) > > Normally, 'freetype2' and 'freetype2-demos' stay in sync date-wise. > What about using the date of a given 'freetype2' commit to get a > corresponding commit in 'freetype2-demos'? > > > 3. I currently run the demos in xvfb ... > > Isn't `xvfb` going to be replaced with `xf86-video-dummy`? > > > ... but this isn't exactly compatible with windows so if you wish > > for the regression test to run there I might need to modify some > > the demos to just dump an image without opening a window > > This might be generally useful, yes. However, I suggest that you > concentrate on having the tests first. It is rather unlikely that > FreeType itself behaves differently on Windows since it is quite > generic C code with only a very small amount of platform-specific > code. In other words, getting screenshots of running FreeType demo > tools on Windows makes sense to catch problems in the GUI code of the > tools but probably not of the FreeType library. For the time being I > think it is sufficient to have Windows compilation tests. > > > 4. I don't see a clear way to get the number of characters in a font > > using any of the existing demos so I might need to add a utility > > for that but for now I just hardcode those values in each font > > test > > Please be careful with terminology: In general, you count the numbers > of *glyphs* in a font. The number of character codes is strongly > dependent on a font's cmaps. So I guess you want to know the number > of glyphs, right? > > `ftdump` shows a 'glyph count' line. Maybe this works for you. > Alternatively, a small tool to explicitly get this value should be > easy to write. > > > 5. I'm not sure ftgrid and etc support all render modes? > > Mhmm, maybe. With `ftgrid`, for example, use keys 'F5' and 'F6' to > cycle through rendering modes. > > Alexei, how do I specify keys like 'F5' with the `-k` command line > option? > > > I'm currently working on the report generation as well as 1/2 and > > hope to have something to show soon. However, to show results it > > would be helpful if someone could point me to a line of code I could > > tweak to subtly break rendering between commits so we can see it > > find faults. > > It's not clear to me what you want to achieve. Please elaborate. > > > Also, if there are already any solutions to my issues with demos > > I've missed please let me know so I don't reinvent the wheel. > > If my explanations above are not sufficient please don't hesitate to > ask more! > > > Werner >
