Re: Testing framework
I reran the azure job a few days ago. https://dev.azure.com/fundies/freetype2/_build/results?buildId=307=logs=3a288725-918b-506e-a9e9-b0c5a06e5dbe=3a288725-918b-506e-a9e9-b0c5a06e5dbe=1bdf8318-1846-519a-f285-c1e66a0a38e5 The regression tests still work. So any issue you are all getting is triggered either from updates to freetype since gsoc ended (such as renaming your repos from freetype2-> freetype) or some difference in your setup to mine that I can't reproduce. My azure job installs a fresh copy of archlinux in a chroot so you should be able to literally follow it step by step on your PC or virtual machine. As for the calls to xvfb, they should be replaced with calling ft-demos programs with arguments to dump the images. I asked if such arguments existed but both my mentor and I weren't aware these arguments existed when I wrote the scripts so I used xvfb. A few students emailed me directly with a "this doesn't work please help" message. However, I cannot help anyone without you giving proper logs and documenting the steps you took to arrive at that error / state. Also, it is best to ask the mailing list directly so you have more eyes on your question. Please forward all your questions to the list directly and I will answer what I can but I will need more information. Thanks, Greg On Sat, Apr 10, 2021 at 11:56 AM Alexei Podtelezhnikov wrote: > > > > On Sat, Apr 10, 2021 at 11:12 AM sarthak bhardwaj > <1sarthakbhard...@gmail.com> wrote: > > Yes, actually I have done it yesterday itself, > > Great! We use these tools to test FreeType. The whole idea of the project is > to automate the process: > compare these or similar images for the current master against some stock > collection from a previous release to detect regressions. > > - gross comparison as md5sum > - average contrast and "brightness" comparison > - blink comparison and/or difference image > > Something along these lines. > > Alexei >
Re: Status of the test framework project
Having it post an image / comment is easy enough. I already do this on some of my github projects. However, as I mentioned savannah is missing tools to do things like this. I will try to make a PR once you have migrated to gitlab or elsewhere but until then I'd rather not waste the effort until you've committed to a more modern git interface. On Mon, Jan 11, 2021 at 1:51 AM Werner LEMBERG wrote: > > > Hello Greg, > > > thanks for chiming in! > > > Running continuous integration on Savannah isn't really an option. > > Yes. > > > If the project moves to gitlab as was discussed months ago. I would > > not be opposed to porting my azure configs to gitlab ones. > > Hopefully, this is done soon. Anurag? > > > However it is not really practical for me to set up servers for your > > organization. The scripts I wrote should run on any Linux machine > > though. > > OK. > > > "Improving the UI" is really vague so it would be helpful to me or > > your future student if you're clearer about what you want upfront. > > For comparison, have a look at LilyPond: If a merge requests gets > submitted, its test suite is run, and the differences to a base line > are automatically provided in quite a nice way (including the > possibility for blink comparison). > > Here is an example MR: > > https://gitlab.com/lilypond/lilypond/-/merge_requests/589 > > and here the artifacts of a successfully executed pipeline job for > online viewing: > > > https://lilypond.gitlab.io/-/lilypond/-/jobs/944871436/artifacts/test-results/index.html > > Normally this link would expire tomorrow and the artifacts removed; > I've pressed the 'Keep' button right now so that they stay as an > example. > > It is also possible to download the artifacts as a bundle: > > https://gitlab.com/lilypond/lilypond/-/merge_requests/589/pipelines > > > I don't know much about bcmp or whatever but switching out the > > comparison tool used in the scripts shouldn't be difficult. > > Yep. > > To summarize: It would be great if your scripts can be adapted and > integrated to the FreeDeskTop instance of gitlab as soon as the > repository together with the issues database has moved. > > > Werner
Re: Status of the test framework project
Running continuous integration on Savannah isn't really an option. If the project moves to gitlab as was discussed months ago. I would not be opposed to porting my azure configs to gitlab ones. However it is not really practical for me to set up servers for your organization. The scripts I wrote should run on any Linux machine though. "Improving the UI" is really vague so it would be helpful to me or your future student if you're clearer about what you want upfront. I don't know much about bcmp or whatever but switching out the comparison tool used in the scripts shouldn't be difficult. On Sun, Jan 10, 2021, 9:44 AM Werner LEMBERG wrote: > > > What is the status of the test framework project? > > Dormant. > > > It was attempted during GSoC 2020 but the submitted code does not > > satisfy one of the design criteria: just work. Several parts of the > > project can be improved. > > I fully agree. Collaboration with the student was difficult, > unfortunately. > > > - Expensive Microsoft Azure nodes can be replaced with local > > GNU/Linux boxes. While the computing requirements of the project > > are moderate, keeping raster data in cloud storage is > > unaffordable. > > > > - The general purpose image comparison can be replaced with more > > finely tuned solutions. The bmpcmp utility that is used by > > Ghostscript developers is a good starting point. > > > > - The UI can be more sophisticated. > > > > - There is no test case collection. > > > > - The project is abandoned by the author. There's no new commits > > after the official end of the project. > > I fully agree with your analysis. > > > If the test framework project is re-offered in 2021, it could be > > done better. > > Yes, probably. This would be the third attempt, then :-) > > AFAICS, what was done during GSoC 2020 looks promising, so we would > need a dedicated student to continue the work. > > By the way, GSoC 2021 will work differently: The available time for > students gets halved; this implies smaller projects. In other words, > what you suggest might indeed be a good solution. > > > Werner > >
Re: upcoming raster changes
I'm not sure what you mean by "blink". I do generate a html page where you can mouse over images to see the differences. Is that what you're referring to? If it's not all my tests just run existing demos and I can tell you where to add new commands if you need them. On Sat, Sep 12, 2020 at 3:31 PM Alexei Podtelezhnikov wrote: > > > > Alexei A Podtelezhnikov, PhD > > > > > > > I asked if there were flags to dump the images at some point > > but I didn't see anything mentioned in the --help. > > I apologize I did not realize that you were working on it besides just > testing the build environment. Have you implemented a blink test too? > > > It should be > > pretty simple to swap out my xvfb command with whatever you want > > though. > > > > On Sat, Sep 12, 2020 at 1:53 PM Alexei Podtelezhnikov > > wrote: > >> > >>> On Sat, Sep 12, 2020 at 12:07 PM Greg Williamson > >>> wrote: > >>> > >>> You should be able to run the tests I made for my gsoc project locally > >>> on any linux machine. There are instructions in CI/readme for running > >>> them > >> > >> Hi Greg, > >> > >> Are you aware of -k option or rather -kPq sequence to dump PNG (eg > >> ftgrid.png) without xvfbRunAndScreenShot? This could be a lot faster > >> with x11 out of the way. > >> The -k option passes the keystroke sequence and can, for example, help > >> you switch to the monochrome mode non-interactively. The changes in > >> question will not be visible in other modes. > >> > >> Thanks, > >> Alexei
Re: upcoming raster changes
I asked if there were flags to dump the images at some point but I didn't see anything mentioned in the --help. It should be pretty simple to swap out my xvfb command with whatever you want though. On Sat, Sep 12, 2020 at 1:53 PM Alexei Podtelezhnikov wrote: > > On Sat, Sep 12, 2020 at 12:07 PM Greg Williamson wrote: > > > > You should be able to run the tests I made for my gsoc project locally > > on any linux machine. There are instructions in CI/readme for running > > them > > Hi Greg, > > Are you aware of -k option or rather -kPq sequence to dump PNG (eg > ftgrid.png) without xvfbRunAndScreenShot? This could be a lot faster > with x11 out of the way. > The -k option passes the keystroke sequence and can, for example, help > you switch to the monochrome mode non-interactively. The changes in > question will not be visible in other modes. > > Thanks, > Alexei
Re: upcoming raster changes
You should be able to run the tests I made for my gsoc project locally on any linux machine. There are instructions in CI/readme for running them On Fri, Sep 11, 2020 at 5:39 PM Nikolaus Waxweiler wrote: > > If only we had a test suite with comparison images, we could see the effect > these patches have immediately!
Re: GSOC Build tests
Ok I've wrapped the readme lines as requested On Sun, Aug 30, 2020 at 4:02 PM Werner LEMBERG wrote: > > > > Ok I believe I was able to commit to savannah but the message is a > > bit cryptic: > > Everything's fine, thanks. > > > I also added a readme as requested in CI/readme.md. > > Thanks. Please reformat the file to make the lines not longer than 78 > characters. > > > Can anyone confirm this worked and tell me how I can link to it in > > my final evaluation? > > Use this > > https://git.savannah.gnu.org/cgit/freetype/freetype2.git/?h=GSoC-2020-greg > > or something similar offered on that page (whatever fits you best). > > > I was also able to fix a few msys build errors but I still cannot > > build demos in msys due it trying to include termios.h which is not > > available. For now, I've temporarily disabled building demos for that > > platform. > > OK. > > > Werner
Re: GSOC Build tests
Ok I believe I was able to commit to savannah but the message is a bit cryptic: [greg@greg-desktop savannah-freetype2]$ git push --set-upstream origin GSoC-2020-greg Enumerating objects: 23, done. Counting objects: 100% (23/23), done. Delta compression using up to 8 threads Compressing objects: 100% (19/19), done. Writing objects: 100% (19/19), 21.39 KiB | 10.69 MiB/s, done. Total 19 (delta 5), reused 0 (delta 0), pack-reused 0 remote: *** no recipients configured so no email will be sent remote: *** for 'refs/heads/GSoC-2020-greg' update None->3f82a109c0f16458b05110beeb2ad443ba0274bd To git.savannah.nongnu.org:/srv/git/freetype/freetype2.git * [new branch] GSoC-2020-greg -> GSoC-2020-greg Branch 'GSoC-2020-greg' set up to track remote branch 'GSoC-2020-greg' from 'origin'. I also added a readme as requested in CI/readme.md. Can anyone confirm this worked and tell me how I can link to it in my final evaluation? I was also able to fix a few msys build errors but I still cannot build demos in msys due it trying to include termios.h which is not available. For now, I've temporarily disabled building demos for that platform. On Sat, Aug 29, 2020 at 10:23 AM Werner LEMBERG wrote: > > > Hello Greg, > > > sorry for the late reply. > > > As, discussed with Werner I have: > > 1) added a simple config file (in CI/ft-tests.config) > > 2) split the tests into smaller chunks > > 3) changed the colors of the diff images to green / red > > 4) added the diff images to the generated comparison page > > 5) fixed the relative paths in the generated comparison page > > This looks good, thanks. However, I miss a README document that > explains where to start... > > > I have also made an attempt to add a fix for the MinGW builds but it > > appears your make install script can't handle window paths so we'll > > need to sed the \ to / or something to fix that build test. > > Please show an error log. Maybe it's easy to fix. > > > If you have any issues or questions about my code I will still be > > checking this email even after GSoC so let me know and I will do my > > best to fix whatever is wrong. > > Great, thanks! > > > In the gitlab migration listing I can talk about integrating this > > before or after you move but I am more familiar with github and can > > only really answer questions on my azure config file and scripts or > > integrating it with github. > > We would be indeed glad if you could assits with CI stuff since my > knowledge is essentially zero. > > > As for my side of the GSoC evaluation, Where do your students > > typically link to for the "showcase" of their project? > > You should put your code into a branch named 'GSoC-2020-greg' (or > something similar) of the FreeType repository. > > > I was never able to successfully push to your repo > > This we have to fix right now. What exactly is the problem? Have you > uploaded an SSH key to Savannah for your account? > > https://savannah.gnu.org/maintenance/SshAccess/ > > Be careful not to add a final newline character that some editors > automatically insert. After this, you should be able to checkout the > FreeType repositories with > > git clone fund...@git.savannah.nongnu.org:/srv/git/freetype/freetype2.git > git clone > fund...@git.savannah.nongnu.org:/srv/git/freetype/freetype2-demos.git > > > Werner
Re: GSOC Build tests
Ok I believe I was able to commit to savannah but the message is a bit cryptic: [greg@greg-desktop savannah-freetype2]$ git push --set-upstream origin GSoC-2020-greg Enumerating objects: 23, done. Counting objects: 100% (23/23), done. Delta compression using up to 8 threads Compressing objects: 100% (19/19), done. Writing objects: 100% (19/19), 21.39 KiB | 10.69 MiB/s, done. Total 19 (delta 5), reused 0 (delta 0), pack-reused 0 remote: *** no recipients configured so no email will be sent remote: *** for 'refs/heads/GSoC-2020-greg' update None->3f82a109c0f16458b05110beeb2ad443ba0274bd To git.savannah.nongnu.org:/srv/git/freetype/freetype2.git * [new branch] GSoC-2020-greg -> GSoC-2020-greg Branch 'GSoC-2020-greg' set up to track remote branch 'GSoC-2020-greg' from 'origin'. I also added a readme as requested in CI/readme.md. Can anyone confirm this worked and tell me how I can link to it in my final evaluation? I was also able to fix a few msys build errors but I still cannot build demos in msys due it trying to include termios.h which is not available. For now, I've temporarily disabled building demos for that platform. On Sat, Aug 29, 2020 at 10:23 AM Werner LEMBERG wrote: > > > Hello Greg, > > > sorry for the late reply. > > > As, discussed with Werner I have: > > 1) added a simple config file (in CI/ft-tests.config) > > 2) split the tests into smaller chunks > > 3) changed the colors of the diff images to green / red > > 4) added the diff images to the generated comparison page > > 5) fixed the relative paths in the generated comparison page > > This looks good, thanks. However, I miss a README document that > explains where to start... > > > I have also made an attempt to add a fix for the MinGW builds but it > > appears your make install script can't handle window paths so we'll > > need to sed the \ to / or something to fix that build test. > > Please show an error log. Maybe it's easy to fix. > > > If you have any issues or questions about my code I will still be > > checking this email even after GSoC so let me know and I will do my > > best to fix whatever is wrong. > > Great, thanks! > > > In the gitlab migration listing I can talk about integrating this > > before or after you move but I am more familiar with github and can > > only really answer questions on my azure config file and scripts or > > integrating it with github. > > We would be indeed glad if you could assits with CI stuff since my > knowledge is essentially zero. > > > As for my side of the GSoC evaluation, Where do your students > > typically link to for the "showcase" of their project? > > You should put your code into a branch named 'GSoC-2020-greg' (or > something similar) of the FreeType repository. > > > I was never able to successfully push to your repo > > This we have to fix right now. What exactly is the problem? Have you > uploaded an SSH key to Savannah for your account? > > https://savannah.gnu.org/maintenance/SshAccess/ > > Be careful not to add a final newline character that some editors > automatically insert. After this, you should be able to checkout the > FreeType repositories with > > git clone fund...@git.savannah.nongnu.org:/srv/git/freetype/freetype2.git > git clone > fund...@git.savannah.nongnu.org:/srv/git/freetype/freetype2-demos.git > > > Werner
Re: GSOC Build tests
As, discussed with Werner I have: 1) added a simple config file (in CI/ft-tests.config) 2) split the tests into smaller chunks 3) changed the colors of the diff images to green / red 4) added the diff images to the generated comparison page 5) fixed the relative paths in the generated comparison page I have also made an attempt to add a fix for the MinGW builds but it appears your make install script can't handle window paths so we'll need to sed the \ to / or something to fix that build test. The current patch is here: https://patch-diff.githubusercontent.com/raw/fundies/freetype2/pull/1.patch You can see the CI in action here: https://dev.azure.com/fundies/freetype2/_build/results?buildId=237=logs=05dc4cca-e891-599c-177e-6a5b81dbe26c=05dc4cca-e891-599c-177e-6a5b81dbe26c You can download an example test run here: https://dev.azure.com/fundies/9eabb07a-6a4d-4b68-b22e-60f9e02c1927/_apis/build/builds/237/artifacts?artifactName=Archlinux%20Regression%20tests%20Test2=6.0&%24format=zip If you have any issues or questions about my code I will still be checking this email even after GSoC so let me know and I will do my best to fix whatever is wrong. In the gitlab migration listing I can talk about integrating this before or after you move but I am more familiar with github and can only really answer questions on my azure config file and scripts or integrating it with github. As for my side of the GSoC evaluation, Where do your students typically link to for the "showcase" of their project? I was never able to successfully push to your repo so I just moved on and worked on my project on github. However, as it's not an official mirror I don't think it'd be appropriate to link there. On Thu, Aug 6, 2020 at 1:40 AM Werner LEMBERG wrote: > > > Hi Greg, > > > sorry for the late response. > > > > [...] if you have no preference I have an easy way to handle configs > > in mind. It won't be the prettiest config file you've ever seen but > > it should minimize the amount of additional code needed. > > Just proceed, please! > > >> Looking around I've found the difference png – what do you think of > >> slightly coloring the differences? And for comparing B/W images, I > >> remember that I've coloured points only in the reference image as > >> green and in the new image as red... > > > > As for the diff image you saw, I am limited to what imagick can do. > > You can see an overview of their image diff modes here: > > http://www.imagemagick.org/Usage/compare/#methods > > Ah, ok. > > > I can use any of these and even multiple of them but the more images > > the larger the zip of course. Please have a peek and let me know > > which modes you think would be most useful to you and I can add them > > to the diffing page. > > Just something with color :-) > > > The highlight/lowlight example on the page is what you mean when > > youre talking about green/red? > > Maybe, I'm not sure. What I did was the following for a comparsion > between two bitmaps A and B: > > present in Apresent in Bcolor > - >0 0 white >1 0 green >0 1 red >1 1 black > > For hovering bitmaps this would be a possibility. For hovering > graymaps I guess it doesn't work. > > > I am/will be working on what we've discussed above and hope to have a > > new patch / commit for you soon. > > Great. Please send an e-mail to the list if you have something to > show. > > > Werner
Re: Gitlab Migration Progress
> Thanks! I suppose this is the correct repo: > https://github.com/fundies/freetype2/tree/greg-gsoc > > BTW Has it been decided that we would be using azure? Gitlab's CI offering > seem pretty good too. > > Regards, > Anurag Yes, that is the correct repo. Azure offers 10 free continuous jobs for open source projects which I think would be silly not to take advantage of. You could port my yaml to gitlab, however there is nothing prohibiting you from using both (or even more CI services). It's not a "this" vs "that" choice. In my other projects I use azure, travis-ci and appveyor all together to distribute the number of free server instances I get. On Thu, Aug 20, 2020 at 2:18 PM Anurag Thakur wrote: > > Hey guys! So, I have almost completed the script for migration. Here is a > Google drive link to see it in action(it's a video): > https://drive.google.com/file/d/102TpqRiFv32PKkPLSASPasj5XCC71Ok1/view?usp=drivesdk > > Please go to gitlab.com/freetype2/issue-migration/issues/40 and tell me if > any further changes are required in the issue format. The only change that's > left from my side is adding an additional comment with link to all the > attached files to every issue. > > Also, everyone please send me your gitlab ID with preferred > position(owner/maintainer/developer/member) so that I can invite you to the > freetype2 group. > > I will be posting any further updates regarding the migration on this thread. > > Please let me know what else is required for migrating fully to gitlab (the > issue migration should be done by tomorrow). > > Regards, > Anurag
Re: GSOC Build tests
> This looks very good, thanks! Indeed, the archive size of almost one > Gigabyte is far too large for practical purposes. We have now to find > solutions in the next month how to refine that. Right now it runs ~12 fonts that have ~3k glyphs each; this is why it's so large. I think the best solution is splitting up runs of the tests to limit it to specific fonts and glyph ranges. That way you can download the smaller subset of tests that fail. > What do you think about using a control file that specifies which > fonts to use, which glyphs in those fonts to use at which sizes, > modes, etc.? The scripts are already basically structured to do this so adding that should be easy. If you could give me an idea of how you would like the config file structured I can write a parser easily enough. > I think some CSS magic to make the results more pleasing would also be > nice, but this is an extra of no particular importance currently. I've never been the best at design. If someone can give me a design mockup or something similar they'd like it to look to though I can make it match fairly easily. > The most important thing IMHO is that you (manually) prepare a subset > of the large report now (not larger than, say, 1MByte) so that other > interested persons can quickly download it to have a look. > Additionally, it might be helpful if you could send some screen shots > to the list. Here is a small subset of the glyphs. I've manually modified one in gimp to show the image diffing. https://drive.google.com/file/d/1Y5MZ1laLK0UrZyCxbYIq3sptFDIfhoSg/view?usp=sharing Here are some images: https://imgur.com/cBTwrDr.png https://imgur.com/uCx2xOI.png https://imgur.com/hPtWmJI.png https://imgur.com/AyRMA92.png On Thu, Jul 30, 2020 at 4:39 AM Werner LEMBERG wrote: > > > I've finished the core of the regression tester. > > Great! > > > You can now run it and generate a html report but you will need a > > few tools installed: imagick, xwd, npm, pretty-diff and xvfb. This > > should all be listed in ft-regression.sh's comments.If you want to > > run it locally make sure you have a ~/test-fonts with .ttf files in > > it. > > Thanks; I will try that in the next few days, hopefully. > > > This is basically what I wanted to accomplish for my GSOC project > > but I can do a lot of the things mentioned in my previous message if > > I pass my evaluation. > > :-) > > > You can download the report it generates from here: > > > https://dev.azure.com/fundies/9eabb07a-6a4d-4b68-b22e-60f9e02c1927/_apis/build/builds/232/artifacts?artifactName=Archlinux%20Regression%20tests=5.1&%24format=zip > > The file is rather large. In the future, I could probably shrink it > > quite a bit by using 7zip instead of regular zip or I could split > > the tests into smaller chunks too. > > This looks very good, thanks! Indeed, the archive size of almost one > Gigabyte is far too large for practical purposes. We have now to find > solutions in the next month how to refine that. > > What do you think about using a control file that specifies which > fonts to use, which glyphs in those fonts to use at which sizes, > modes, etc.? > > I think some CSS magic to make the results more pleasing would also be > nice, but this is an extra of no particular importance currently. > > > The report doesn't demonstrate the image comparison because there > > are obviously no regressions between my commit and master. However, > > if there is one it will generate a page where you can see the > > differences between images on mouse over. For text files it > > generates a diff report using pretty-diff. You can see this in the > > freetype-bench comparisons. > > I *have* to see the image comparison live :-) When testing this > locally, I'll try to temporarily introduce a bug – since this depends > on the selected fonts I also have to do some trial and error to find a > good spot to do that. > > > As I've said this should be mostly working now. I also believe I've > > commented the source pretty thoroughly. I've also moved the scripts > > to a subdir as requested. If there are any changes / improvements > > you would like to see to the reports or scripts please let me know. > > If not, I have listed several things I would like to do to polish it > > in my previous message. > > The most important thing IMHO is that you (manually) prepare a subset > of the large report now (not larger than, say, 1MByte) so that other > interested persons can quickly download it to have a look. > Additionally, it might be helpful if you could send some screen shots > to the list. > > > Werner >
Re: GSOC Build tests
I've finished the core of the regression tester. You can now run it and generate a html report but you will need a few tools installed: imagick, xwd, npm, pretty-diff and xvfb. This should all be listed in ft-regression.sh's comments.If you want to run it locally make sure you have a ~/test-fonts with .ttf files in it. You will also need freetype2 and freetype2-demos sources next to each other. You can run it locally by calling: ./CI/ft-regression.sh eg: ./CI/ft-regression.sh HEAD~1 from inside your freetype2 directory. It should generate an easy to digest report to /tmp/ft-tests/index.html. This is basically what I wanted to accomplish for my GSOC project but I can do a lot of the things mentioned in my previous message if I pass my evaluation. I've also got the testing apparatus running on azure pipelines with the other tests here: https://dev.azure.com/fundies/freetype2/_build/results?buildId=232=results Ignore the mingw build failures as they seem to be in the midst of a regime change and their new signed key databases aren't in sync with azure. You can download the report it generates from here: https://dev.azure.com/fundies/9eabb07a-6a4d-4b68-b22e-60f9e02c1927/_apis/build/builds/232/artifacts?artifactName=Archlinux%20Regression%20tests=5.1&%24format=zip The file is rather large. In the future, I could probably shrink it quite a bit by using 7zip instead of regular zip or I could split the tests into smaller chunks too. The report doesn't demonstrate the image comparison because there are obviously no regressions between my commit and master. However, if there is one it will generate a page where you can see the differences between images on mouse over. For text files it generates a diff report using pretty-diff. You can see this in the freetype-bench comparisons. As I've said this should be mostly working now. I also believe I've commented the source pretty thoroughly. I've also moved the scripts to a subdir as requested. If there are any changes / improvements you would like to see to the reports or scripts please let me know. If not, I have listed several things I would like to do to polish it in my previous message. Here is my patchset: https://patch-diff.githubusercontent.com/raw/fundies/freetype2/pull/1.patch On Tue, Jul 21, 2020 at 12:23 AM Werner LEMBERG wrote: > > Hello Greg, > > > > figured out the cause. I have however been working on this as much > > as time allows. I've mostly hashed out the scripts to run regression > > tests using demos here: > > > https://patch-diff.githubusercontent.com/raw/fundies/freetype2/pull/1.patch > > [...] > > Some comments. > > * There are too much top-level files. Please move them to > subdirectories. > > * Please submit patches to the list that are generic enough to be > included into 'master'. This affects the CMake stuff, for example. > > * If possible, don't use lines longer than 80 characters. > > * Please avoid trailing whitespace. > > > . The scripts still need a bit of work. I've only just started the > > bits that generate a html report for devs to inspect and I need to > > hook it into azure pipelines. > > My problem is that I have zero knowledge of azure. So please... > > > A couple other issues I want/plan to address are: > > > > 1. I need to comment the scripts better > > ... do that in the first place. > > > 2. I need to add a script argument to specify the demos commit as > >not all commits of freetype2 are compatible with freetype2-demos > >(in fact this broke on me over the past couple weeks) > > Normally, 'freetype2' and 'freetype2-demos' stay in sync date-wise. > What about using the date of a given 'freetype2' commit to get a > corresponding commit in 'freetype2-demos'? > > > 3. I currently run the demos in xvfb ... > > Isn't `xvfb` going to be replaced with `xf86-video-dummy`? > > >... but this isn't exactly compatible with windows so if you wish > >for the regression test to run there I might need to modify some > >the demos to just dump an image without opening a window > > This might be generally useful, yes. However, I suggest that you > concentrate on having the tests first. It is rather unlikely that > FreeType itself behaves differently on Windows since it is quite > generic C code with only a very small amount of platform-specific > code. In other words, getting screenshots of running FreeType demo > tools on Windows makes sense to catch problems in the GUI code of the > tools but probably not of the FreeType library. For the time being I > think it is sufficient to have Windows compilation tests. > > > 4. I don't see a clear way to get the number of characters in a font > >using any of the existing demos so I might need to add a utility > >for that but for now I just hardcode those values in each font > >test > > Please be careful with terminology: In general, you count the numbers > of *glyphs* in a font. The number of character codes is strongly > dependent on a font's
Re: GSOC Build tests
Hi, sorry for lack of responses. I have been very busy with final projects for my summer classes and to top it off, some of the ram (or slots on the motherboard) in my computer died and my computers were super wonky until I figured out the cause. I have however been working on this as much as time allows. I've mostly hashed out the scripts to run regression tests using demos here: https://patch-diff.githubusercontent.com/raw/fundies/freetype2/pull/1.patch . The scripts still need a bit of work. I've only just started the bits that generate a html report for devs to inspect and I need to hook it into azure pipelines. A couple other issues I want/plan to address are: 1. I need to comment the scripts better 2. I need to add a script argument to specify the demos commit as not all commits of freetype2 are compatible with freetype2-demos (in fact this broke on me over the past couple weeks) 3. I currently run the demos in xvfb but this isn't exactly compatible with windows so if you wish for the regression test to run there I might need to modify some the demos to just dump an image without opening a window 4. I don't see a clear way to get the number of characters in a font using any of the existing demos so I might need to add a utility for that but for now I just hardcode those values in each font test 5. I'm not sure ftgrid and etc support all render modes? I'm currently working on the report generation as well as 1/2 and hope to have something to show soon. However, to show results it would be helpful if someone could point me to a line of code I could tweak to subtly break rendering between commits so we can see it find faults. Also, if there are already any solutions to my issues with demos I've missed please let me know so I don't reinvent the wheel.
Re: GSOC Build tests
Over the past couple weeks I've converted the build tests to use autotools for the three desktop OSes. I've managed to leave the cmake tests intact too. I also now have demos building against the freetype version pulled in the CI. I wasted quite a bit of time trying to get autotools to work with MSVC to no avail so if anyone has had success with it please let me know. You can see a preview of the updated build tests here: https://dev.azure.com/fundies/freetype2/_build/results?buildId=195. However, the mingw builds are currently failing because it seems their repo is in process of updating but I promise that it worked before :). I've tried to structure the CI's yaml config in a way that it will be easy to add tests for other build systems for the future when you switch to meson or whatever. I think with the current configuration I'm hitting most build configurations but we can always add more later. Over the coming week I plan to write some functionality tests. I believe my best course of action for this is just to use the existing freetype demo programs in tandem with some bash scripting to check for inconsistencies in various fonts between commits. I will hope to have some tests and some of the UI for manual inspection of failures ready by early next week. On Sun, Jun 7, 2020 at 6:46 AM Alexei Podtelezhnikov wrote: > > >> I would also like to test freetype-demos but I'm not sure if those > >> are only valid on desktop configurations. > > > > Yes, it might be problematic to test the demo programs that uses a > > GUI. Maybe (some of) the command line tools will work. > > All GUI programs are compiled with a GUIless batch driver. In ftview, > ftstring, ftggrid it is even usable with “-k Pq” for example. The other GUI > progs are ftgamma, ftmulti, and ftdiff. There are and console ftdum, > ftbench and ttdebug in good shape. The rest are not much used and rather > obsolete. > >
Re: GSOC Build tests
So autotools is the current build system but that will only work on linux and mingw correct? I can certainly check freetype builds with autotools on those platforms but I don't think it's compatible with msvc & xcode. I can't really check meson if it hasn't been checked in yet afaict. I'm not the biggest fan of cmake and your cmakelists does seem to have a few issues but I think it's far more widely used than meson. I see many projects and even IDEs such as visual studio / qtrceator / codelite and more have built-in cmake support whereas meson is more of a niche. I don't mind adding build tests for meson if that's the route you choose but I think fixing your cmake issues is more practical. As for demos, I only mean to check that they compile against the current freetype. I assume however these demos will not build on iOS so I won't try testing demos there. If I'm incorrect on that let me know and I'll adjust the tests accordingly. As I go forward and start functionality tests I will likely write a standalone program that links freetype and google test using cmake and as I am most familiar with cmake but it shouldn't be much work to convert it to whatever build system you decide on later. Thanks for instructions on git. I typically use github with my login directly and not ssh keys so I wasn't familiar with that process. I'll get that setup asap and push my commits there. On Sat, Jun 6, 2020 at 9:31 AM Werner LEMBERG wrote: > > Hello Greg, > > > > So over the week I started writing continuous integration build > > tests for several platforms. You can preview them here: > > > https://dev.azure.com/fundies/freetype2/_build/results?buildId=146=results > > this looks great! > > > Please let me know if there are any additional compilers, operating > > systems or configurations you would like tested. > > Maybe Android? I'm not sure whether this is necessary, given that it > is basically a Unix system. > > > Also, if you would like any other build system tested other than > > cmake tested. > > Actually, the 'official' build system is > > ./configure > make > make install > > The cmake stuff is contributed code not maintained by the FreeType > team. We probably will migrate to another build system like Meson in > the near future; you can find some discussion in this thread: > > > https://lists.nongnu.org/archive/html/freetype-devel/2020-05/msg00103.html > > > I would also like to test freetype-demos but I'm not sure if those > > are only valid on desktop configurations. > > Yes, it might be problematic to test the demo programs that uses a > GUI. Maybe (some of) the command line tools will work. > > > I did have some issues with Mac/iOS builds as your repo seems to > > have an iOS.cmake that is 6 years old and incompatible with modern > > xcode and I had so set a blank signing key for OS X. I found a > > newer iOS.cmake to replace your current one and it seems to work > > however it does not currently bundle the binaries in the packaging > > stage. Please let me know if you have better solution for either of > > these. > > Maybe other people can chime in who have more experience with cmake. > > > Lastly, I'm not familiar with how to commit to your repo I only get > > "fatal: remote error: access denied or repository not exported: > > /freetype/freetype2.git" without ever being prompted for a login. > > This is an issue that must be fixed. Have you registered your SSH > public key? > > https://savannah.gnu.org/maintenance/SshAccess/ > > (be careful not to add a final, additional newline if you submit the > key). > > After doing the above, you should be able to check out read/write > versions of the FreeType git repositories as > > git clone fund...@git.sv.gnu.org/srv/git/freetype/freetype2.git > git clone fund...@git.sv.gnu.org/srv/git/freetype/freetype2-demos.git > > > So instead here is a patch: > > > https://github.com/fundies/freetype2/commit/564f8d163fbed71cccd1a9ea5733e28da82b7d7e.patch > > I hope you will be able to check this into your personal branch by > yourself soon :-) > > > Werner >
GSOC Build tests
So over the week I started writing continuous integration build tests for several platforms. You can preview them here: https://dev.azure.com/fundies/freetype2/_build/results?buildId=146=results Builds are uploaded here: https://dev.azure.com/fundies/freetype2/_build/results?buildId=146=artifacts=publishedArtifacts Please let me know if there are any additional compilers, operating systems or configurations you would like tested. Also, if you would like any other build system tested other than cmake tested. I would also like to test freetype-demos but I'm not sure if those are only valid on desktop configurations. I did have some issues with Mac/iOS builds as your repo seems to have an iOS.cmake that is 6 years old and incompatible with modern xcode and I had so set a blank signing key for OS X. I found a newer iOS.cmake to replace your current one and it seems to work however it does not currently bundle the binaries in the packaging stage. Please let me know if you have better solution for either of these. Lastly, I'm not familiar with how to commit to your repo I only get "fatal: remote error: access denied or repository not exported: /freetype/freetype2.git" without ever being prompted for a login. So instead here is a patch: https://github.com/fundies/freetype2/commit/564f8d163fbed71cccd1a9ea5733e28da82b7d7e.patch
GSOC
Hi. There appears to be some issue with my school's awful email service and your mailing list so I've created this account in hopes I can better get/send messages. I did reply to werner multiple times but I guess they were missed. I would've much preferred irc or discord but I'll try my best to make this work. I am still on track to start on the first. My plan is to write a basic compilation test and render test for one font to start then build upon that. I could use some suggestions on types of render modes and types of fonts to test as well any common pitfalls you see in commits. However, as I said, I want to start with the most basic test first and then build on that.
