Re: Testing framework

[Alexei Podtelezhnikov]

On Sat, Apr 10, 2021 at 11:12 AM sarthak bhardwaj <
1sarthakbhard...@gmail.com> wrote:
> Yes, actually I have done it yesterday itself,

Great! We use these tools to test FreeType. The whole idea of the project
is to automate the process:
compare these or similar images for the current master against some stock
collection from a previous release to detect regressions.

- gross comparison as md5sum
- average contrast and "brightness" comparison
- blink comparison and/or difference image

Something along these lines.

Alexei

Re: Testing framework

[sarthak bhardwaj]

> Hi Alexei,
>
> Extremely sorry for replying very late,  actually was traveling back from
college, as in our country once again school and colleges are closed due to
the second wave of COVID-19.

Do you have ftview and ftgrid working, either installed from your
> distribution package or compiled? I want to make sure that you tried
> it and know how they work.
>

Yes, actually I have done it yesterday itself, sorry for not informing you
as I have stated the reason I was very busy, and it is working fine for me,
capturing ss from [P] and aborting the process with [esc] key or [q]
and.., and the snapshot is getting saved in the freetype-demos folder
itself naming ftview.png and ftgrid.png in parallel to the command that has
executed, and every time these are updating whenever I run ftview or ftgrid
respectively.
I am sharing the generated pngs...
[image: ftgrid.png]

[image: ftview.png]

Re: Testing framework

[Alexei Podtelezhnikov]

Hi Sarthak,

Do you have ftview and ftgrid working, either installed from your
distribution package or compiled? I want to make sure that you tried
it and know how they work.

On Sat, Apr 10, 2021 at 3:14 AM sarthak bhardwaj
<1sarthakbhard...@gmail.com> wrote:
> I am sorry for the confusion, I mean to say that as Alexei have suggested us 
> to run it directly ,by not using Xvfb and capturing the screenshot by 
> pressing simply [P], that is giving us the .png file saved in same directory.

It is much faster to use ftview and ftgrid in batch mode, which is
invoked by -k followed by keypress sequence ending with q, for
example, -kPq. This way you do not have to waste time creating a
window on the screen and using Xvfb. It is also platform independent..

But again, have you seen how ftview and ftgrid work yet?

Alexei

Re: Testing framework

[Werner LEMBERG]

> I mean to say that as Alexei have suggested us to run it directly,
> by not using Xvfb and capturing the screenshot by pressing simply
> [P], that is giving us the .png file saved in same directory.  I
> think this is what you use to test it in a manual way.

Ah, ok, understood.  Of course, this is just a temporary measure so
that you can make it work in case Xvfb is missing.

> my question is that as the goal is to have a CI script for the
> gitlab instance of 'freetype.org' thats gets called automatically
> and will process the whole thing of checking graphical difference.

IMHO, it would be helpful to find a solution that is as
platform-agnostic as possible.  However, this is not a prerequisite,
since in the CI setup you can usually select an OS.  If the test
comparisons only work under Linux (because it's the simplest solution,
the quickest, etc.), so be it.

> So do we need to follow up with the same technique that is by
> running it directly and capturing screenshots manually all the time
> or we need to have some changes in xvfbRunAndScreenShot itself to
> make it more efficient, or do we need to swap out the xvfb commands
> with some other taking xvfb out of the way..

Whatever fits best.  Doing this investigation is part of the GSoC
project :-)


Werner

Re: Testing framework

[sarthak bhardwaj]

>
> > but still confused over the thing that whether we have to follow the
> > traditional method all the time i.e. xvfbRunAndScreenShot, and check
> > some enhancement in it itself, or we have to do it manually all the
> > time or student need to figure it out by themselves a suitable
> > substitute of it?
>
> I don't understand what you want to say.  What do you mean with
> 'manually'?  What is the 'traditional method'?  Note that my
> misunderstanding might be caused by language barriers, so please
> elaborate.
>
I am sorry for the confusion, I mean to say that as Alexei have suggested
us to run it directly ,by not using Xvfb and capturing the screenshot by
pressing simply [P], that is giving us the .png file saved in same
directory.
 I think this is what you use to test it in a manual way.

my question is that as the goal is to have a CI script for the gitlab
instance of 'freetype.org' thats gets called automatically and will process
the whole thing of checking graphical difference.
So do we need to follow up with the same technique that is by running it
directly and capturing screenshots manually all the time or we need to have
some changes in xvfbRunAndScreenShot itself to make it more efficient, or
do we need to swap out the xvfb commands with some other taking xvfb out of
the way..

>
>
>

Re: Testing framework

[Werner LEMBERG]

> but still confused over the thing that whether we have to follow the
> traditional method all the time i.e. xvfbRunAndScreenShot, and check
> some enhancement in it itself, or we have to do it manually all the
> time or student need to figure it out by themselves a suitable
> substitute of it?

I don't understand what you want to say.  What do you mean with
'manually'?  What is the 'traditional method'?  Note that my
misunderstanding might be caused by language barriers, so please
elaborate.

As mentioned earlier, there might be (documentation) deficiencies in
the stuff that is already available, and this should be rectified and
improved as part of GSoC.

The goal is to have a CI script (or a set of scripts) for the gitlab
instance at 'freedesktop.org' that gets called automatically if a new
commit has been added to the FreeType repository, and that checks
whether there are graphical differences in the rendering output, using
a set of demo fonts (or rather, small subsets of such fonts), which
should be created, too, together with tools that makes it easy to add
fonts to the set.[*]


Werner


[*] Such tools should *not* be written from scratch!  Instead,
existing stuff like the AFDKO
(http://adobe-type-tools.github.io/afdko/AFDKO-Overview.html)
fonttools (https://github.com/fonttools/fonttools) should be used
in scripts or something similar.

Re: Testing framework

[sarthak bhardwaj]

Hi Werner and Alexei,


> It's a good starting point to have a working setup.  To improve it
> is already part of GSoC :-)
>
Ok got it, but still confused over the thing that whether we have to follow
the traditional method all the time i.e.  xvfbRunAndScreenShot, and check
some enhancement  in it itself,
or we have to do it manually all the time or student need to figure it out
by themselves a suitable substitute of it?
Because I think this project is all about automating all the process
therefore it seems more likely to me that we should not use manual aspect
all the time, rather will find a way that will work well...
please correct of I am getting it wrong, and do share your valuable
instructions,
   Thanks in advance

Regards,
Sarthak