Re: Testing framework

[Greg Williamson]

I reran the azure job a few days ago.
https://dev.azure.com/fundies/freetype2/_build/results?buildId=307=logs=3a288725-918b-506e-a9e9-b0c5a06e5dbe=3a288725-918b-506e-a9e9-b0c5a06e5dbe=1bdf8318-1846-519a-f285-c1e66a0a38e5
The regression tests still work. So any issue you are all getting is
triggered either from updates to freetype since  gsoc ended (such as
renaming your repos from freetype2-> freetype) or some difference in
your setup to mine that I can't reproduce. My azure job installs a
fresh copy of archlinux in a chroot so you should be able to literally
follow it step by step on your PC or virtual machine. As for the calls
to xvfb, they should be replaced with calling ft-demos programs with
arguments to dump the images. I asked if such arguments existed but
both my mentor and I weren't aware these arguments existed when I
wrote the scripts so I used xvfb. A few students emailed me directly
with a "this doesn't work please help" message. However, I cannot help
anyone without you giving proper logs and documenting the steps you
took to arrive at that error / state. Also, it is best to ask the
mailing list directly so you have more eyes on your question. Please
forward all your questions to the list directly and I will answer what
I can but I will need more information.

Thanks, Greg

On Sat, Apr 10, 2021 at 11:56 AM Alexei Podtelezhnikov
 wrote:
>
>
>
> On Sat, Apr 10, 2021 at 11:12 AM sarthak bhardwaj 
> <1sarthakbhard...@gmail.com> wrote:
> > Yes, actually I have done it yesterday itself,
>
> Great! We use these tools to test FreeType. The whole idea of the project is 
> to automate the process:
> compare these or similar images for the current master against some stock 
> collection from a previous release to detect regressions.
>
> - gross comparison as md5sum
> - average contrast and "brightness" comparison
> - blink comparison and/or difference image
>
> Something along these lines.
>
> Alexei
>

Re: Testing framework

[Alexei Podtelezhnikov]

On Sat, Apr 10, 2021 at 11:12 AM sarthak bhardwaj <
1sarthakbhard...@gmail.com> wrote:
> Yes, actually I have done it yesterday itself,

Great! We use these tools to test FreeType. The whole idea of the project
is to automate the process:
compare these or similar images for the current master against some stock
collection from a previous release to detect regressions.

- gross comparison as md5sum
- average contrast and "brightness" comparison
- blink comparison and/or difference image

Something along these lines.

Alexei

Re: Testing framework

[sarthak bhardwaj]

> Hi Alexei,
>
> Extremely sorry for replying very late,  actually was traveling back from
college, as in our country once again school and colleges are closed due to
the second wave of COVID-19.

Do you have ftview and ftgrid working, either installed from your
> distribution package or compiled? I want to make sure that you tried
> it and know how they work.
>

Yes, actually I have done it yesterday itself, sorry for not informing you
as I have stated the reason I was very busy, and it is working fine for me,
capturing ss from [P] and aborting the process with [esc] key or [q]
and.., and the snapshot is getting saved in the freetype-demos folder
itself naming ftview.png and ftgrid.png in parallel to the command that has
executed, and every time these are updating whenever I run ftview or ftgrid
respectively.
I am sharing the generated pngs...
[image: ftgrid.png]

[image: ftview.png]

Re: Testing framework

[Alexei Podtelezhnikov]

Hi Sarthak,

Do you have ftview and ftgrid working, either installed from your
distribution package or compiled? I want to make sure that you tried
it and know how they work.

On Sat, Apr 10, 2021 at 3:14 AM sarthak bhardwaj
<1sarthakbhard...@gmail.com> wrote:
> I am sorry for the confusion, I mean to say that as Alexei have suggested us 
> to run it directly ,by not using Xvfb and capturing the screenshot by 
> pressing simply [P], that is giving us the .png file saved in same directory.

It is much faster to use ftview and ftgrid in batch mode, which is
invoked by -k followed by keypress sequence ending with q, for
example, -kPq. This way you do not have to waste time creating a
window on the screen and using Xvfb. It is also platform independent..

But again, have you seen how ftview and ftgrid work yet?

Alexei

Re: Testing framework

[Werner LEMBERG]

> I mean to say that as Alexei have suggested us to run it directly,
> by not using Xvfb and capturing the screenshot by pressing simply
> [P], that is giving us the .png file saved in same directory.  I
> think this is what you use to test it in a manual way.

Ah, ok, understood.  Of course, this is just a temporary measure so
that you can make it work in case Xvfb is missing.

> my question is that as the goal is to have a CI script for the
> gitlab instance of 'freetype.org' thats gets called automatically
> and will process the whole thing of checking graphical difference.

IMHO, it would be helpful to find a solution that is as
platform-agnostic as possible.  However, this is not a prerequisite,
since in the CI setup you can usually select an OS.  If the test
comparisons only work under Linux (because it's the simplest solution,
the quickest, etc.), so be it.

> So do we need to follow up with the same technique that is by
> running it directly and capturing screenshots manually all the time
> or we need to have some changes in xvfbRunAndScreenShot itself to
> make it more efficient, or do we need to swap out the xvfb commands
> with some other taking xvfb out of the way..

Whatever fits best.  Doing this investigation is part of the GSoC
project :-)


Werner

Re: Testing framework

[sarthak bhardwaj]

>
> > but still confused over the thing that whether we have to follow the
> > traditional method all the time i.e. xvfbRunAndScreenShot, and check
> > some enhancement in it itself, or we have to do it manually all the
> > time or student need to figure it out by themselves a suitable
> > substitute of it?
>
> I don't understand what you want to say.  What do you mean with
> 'manually'?  What is the 'traditional method'?  Note that my
> misunderstanding might be caused by language barriers, so please
> elaborate.
>
I am sorry for the confusion, I mean to say that as Alexei have suggested
us to run it directly ,by not using Xvfb and capturing the screenshot by
pressing simply [P], that is giving us the .png file saved in same
directory.
 I think this is what you use to test it in a manual way.

my question is that as the goal is to have a CI script for the gitlab
instance of 'freetype.org' thats gets called automatically and will process
the whole thing of checking graphical difference.
So do we need to follow up with the same technique that is by running it
directly and capturing screenshots manually all the time or we need to have
some changes in xvfbRunAndScreenShot itself to make it more efficient, or
do we need to swap out the xvfb commands with some other taking xvfb out of
the way..

>
>
>

Re: Testing framework

[Werner LEMBERG]

> but still confused over the thing that whether we have to follow the
> traditional method all the time i.e. xvfbRunAndScreenShot, and check
> some enhancement in it itself, or we have to do it manually all the
> time or student need to figure it out by themselves a suitable
> substitute of it?

I don't understand what you want to say.  What do you mean with
'manually'?  What is the 'traditional method'?  Note that my
misunderstanding might be caused by language barriers, so please
elaborate.

As mentioned earlier, there might be (documentation) deficiencies in
the stuff that is already available, and this should be rectified and
improved as part of GSoC.

The goal is to have a CI script (or a set of scripts) for the gitlab
instance at 'freedesktop.org' that gets called automatically if a new
commit has been added to the FreeType repository, and that checks
whether there are graphical differences in the rendering output, using
a set of demo fonts (or rather, small subsets of such fonts), which
should be created, too, together with tools that makes it easy to add
fonts to the set.[*]


Werner


[*] Such tools should *not* be written from scratch!  Instead,
existing stuff like the AFDKO
(http://adobe-type-tools.github.io/afdko/AFDKO-Overview.html)
fonttools (https://github.com/fonttools/fonttools) should be used
in scripts or something similar.

Re: Testing framework

[sarthak bhardwaj]

Hi Werner and Alexei,


> It's a good starting point to have a working setup.  To improve it
> is already part of GSoC :-)
>
Ok got it, but still confused over the thing that whether we have to follow
the traditional method all the time i.e.  xvfbRunAndScreenShot, and check
some enhancement  in it itself,
or we have to do it manually all the time or student need to figure it out
by themselves a suitable substitute of it?
Because I think this project is all about automating all the process
therefore it seems more likely to me that we should not use manual aspect
all the time, rather will find a way that will work well...
please correct of I am getting it wrong, and do share your valuable
instructions,
   Thanks in advance

Regards,
Sarthak

Re: testing framework GSOC

[Werner LEMBERG]

>> It is not clear what problems arise for you because of the local
>> vs. remote thing.  Please elaborate.  Maybe I don't understand you
>> correctly.
> 
> When I am trying to run it locally it simply generates blank html
> report of the comparison and blank images, i am doing everything
> what there is specified in the project and what have described,
> [...]

This basically means that the documentation is not sufficient, I
guess.

> at first when i haven't changed the filename to freetypte2 it was
> not even processing and showing errors but now it processing well
> but generating only blank pages.

Ah, the renaming of the git repository from 'freetype2' to 'freetype'
(and 'freetype2-demos' to 'freetype-demos') and thus the names of the
associated directories happened after last year's GSoC, which means
the scripts have to be adapted accordingly.

> I need some help and i request someone to try this out locally on
> your end, and tell me is it working because i have tried several
> times it is not working for me...

I hope it works for you now.  In general, such broad messages like 'it
doesn't work for me inspite of following the instructions to the
letter' doesn't allow us to help you at all.  It is necessary that you
describe your setup and procedure as detailed as possible so that we
can follow you closely.  This includes build log files, environment
settings (if necessary), directory names, setup etc., etc.  In most
cases, the cause of the problem is a typo or a tiny missing detail
that is near to impossible to catch otherwise.


Werner

Re: Testing framework

[Werner LEMBERG]

> inside test-font.sh, there is a function written
> xvfbRunAndScreenShot, it will show error and only blank png's will
> be generated, as far as there is no ../freetype2-demos/bin/ftgrid
> directory found in our demos program, and there is no branch in
> demos for GSoC's Gregs works.  It will only work when ftgrid file
> will get merge to the demos repo.

OK, thanks for the information.

> I have found the solution and is running that way but if it is
> working without the ftgrid file please let me know the way.

It's a good starting point to have a working setup.  To improve it
is already part of GSoC :-)


Werner

Re: testing framework GSOC

[Alexei Podtelezhnikov]

Hi Aman,

Forget about Xvfb. Compile freetype-demos (ftview, and ftgrid) and run them
directly. Try making screenshot by pressing [P] or using -kPq  switch.

Alexei

On Thu, Apr 8, 2021 at 3:25 AM Aman Kapoor 
wrote:

> Thanks Sarthak for the help
>
>> It is not clear what problems arise for you because of the local
>> vs. remote thing.  Please elaborate.  Maybe I don't understand you
>> correctly.
>>
>
> When I am trying to run it locally it simply generates blank html report
> of the comparison and blank images, i am doing everything what there is
> specified in the project and what have described, at first when i haven't
> changed the filename to freetypte2 it was not even processing and showing
> errors but now it processing well but generating only blank pages. And i am
> not getting the differences.
> I need some help and i request someone to try this out locally on
> your end, and tell me is it working because i have tried several times it
> is not working for me...
>
>>

-- 
Alexei A. Podtelezhnikov, PhD

Re: Testing framework

[Alexei Podtelezhnikov]

Hi Sarthak

I suggest using ftgrid or ftview executables directly without
xvfbRunAndScreenShot. They have built-in capability to save PNG
screenshots by pressing [P] or using -k[blah-blah-blah]Pq switch.
Please compile freetype-demos in a parallel folder and try it.

Best,
Alexei

On Thu, Apr 8, 2021 at 8:25 AM sarthak bhardwaj
<1sarthakbhard...@gmail.com> wrote:
>
> hello mentors,
> I had found the same problem as Aman is facing some days before actually, I 
> forgot to intimate about this thing earlier,
> inside test-font.sh, there is a function written xvfbRunAndScreenShot, it 
> will show error and only blank png's will be generated, as far as there is no 
>  ../freetype2-demos/bin/ftgrid directory found in our demos program, and 
> there is no branch in demos for GSoC's Gregs works. It will only work when 
> ftgrid file will get merge to the demos repo.
> I have found the solution and is running that way but if it is working 
> without the ftgrid file please let me know the way.
>
> thanks,



-- 
Alexei A. Podtelezhnikov, PhD

Re: testing framework GSOC

[Aman Kapoor]

>
> Thanks Sarthak for the help

> It is not clear what problems arise for you because of the local
> vs. remote thing.  Please elaborate.  Maybe I don't understand you
> correctly.
>

When I am trying to run it locally it simply generates blank html report of
the comparison and blank images, i am doing everything what there is
specified in the project and what have described, at first when i haven't
changed the filename to freetypte2 it was not even processing and showing
errors but now it processing well but generating only blank pages. And i am
not getting the differences.
I need some help and i request someone to try this out locally on your end,
and tell me is it working because i have tried several times it is
not working for me...

>

Re: testing framework GSOC

[Werner LEMBERG]

Hello Aman,


> i wanna ask that can we run this framework in any other way than
> running locally or not?

The idea is to have this run remotely eventually, using the CI
capabilities of the gitlab instance on freedesktop.org (or maybe
something else in case this isn't sufficient for whatever reason) to
automatically generate HTML pages with images.

However, it doesn't make a big difference: The CI stuff has to be
tested and prepared locally; to do debugging is probably a nightmare
otherwise.

So it boils down whether it is ok that extra prerequisites are
required, and the answer is yes.

Here is an example MR (merge request) for the LilyPond typesetter that
I've already sent some months ago to the list:

  https://gitlab.com/lilypond/lilypond/-/merge_requests/589

and here are the artifacts of a successfully executed pipeline job for
online viewing (doing `make test` for the MR and comparing it to a
baseline's `make test`):

  
https://lilypond.gitlab.io/-/lilypond/-/jobs/944871436/artifacts/test-results/index.html

Whether you say `make test` on your computer or whether the CI engine
does `make test`, there isn't a difference.  In both cases HTML files
are generated.

> Please do share the process because i got nothing related to this
> related on readme..  and is process of running locally is written
> clearly in the documentation because there is difference between
> instructions given in readme and report given to mailing list by
> Greg Williamson.

It is not clear what problems arise for you because of the local
vs. remote thing.  Please elaborate.  Maybe I don't understand you
correctly.


Werner

Re: testing framework GSOC

[Aman Kapoor]

Hi all,

 i wanna ask that can we run this framework in any other way than running
locally or not? Please do share the process because i got nothing related
to this related on readme..and is process of running locally is written
clearly in the documentation because there is difference between
instructions given in readme and report given to mailing list by Greg
Williamson.

And Alexei can you tell that what is the thing you want to improve in image
generation and what is inside the demo programs? That can be useful

Thanks
Aman Kapoor

>
>>
>>
>>