Re: CI+Coverity (was: Re: Working on CI job that runs automated tests)

[Christian Ridderström]

On 7 March 2017 at 11:23, Jean-Marc Lasgouttes  wrote:

> Hi Christian,
>
> The next natural step could be to use your CI stuff to update our coverity
> builds:
> https://scan.coverity.com/projects/4164
>
> Currently the build are submitted by hand by Liviu or recently by myself.
> There is plenty of information on the coverity site on how to automate
> pushing builds. As I understand it, we are entitled to on build per day.
>
> I sent an invite to you, so that you can take a look at the situation.
> FWIW, the configure flags I used are
> ./configure --without-included-boost  --disable-stdlib-debug
>
> The idea is to skip as much code that is not ours as possible.
>

Hi,

I've looked a little at coverity / scan.coverity now and read e.g. the FAQ,
but I'm not clear on how it's supposed to work.
Are you (and Liviu) running coverity locally and then uploading the results
to scan.coverity.com?

Could you please point me to the documentation on the coverity site, I must
be dense but I'm not finding the right info.

Regards,
Christian

PS
I suspect I'll have to become admin at scan.coverity in order to access a
token, in order to be able to upload results from a CI job. (Or you send me
the token off-list)

CI+Coverity (was: Re: Working on CI job that runs automated tests)

[Jean-Marc Lasgouttes]

Le 27/02/2017 à 08:56, Christian Ridderström a écrit :

Hi,

This is a CI job I'm working on that's intended to run automated tests:


Hi Christian,

The next natural step could be to use your CI stuff to update our 
coverity builds:

https://scan.coverity.com/projects/4164

Currently the build are submitted by hand by Liviu or recently by 
myself. There is plenty of information on the coverity site on how to 
automate pushing builds. As I understand it, we are entitled to on build 
per day.


I sent an invite to you, so that you can take a look at the situation. 
FWIW, the configure flags I used are

./configure --without-included-boost  --disable-stdlib-debug

The idea is to skip as much code that is not ours as possible.

JMarc

Re: Working on CI job that runs automated tests

[Scott Kostyshak]

On Tue, Feb 28, 2017 at 09:31:33AM +0100, Kornel Benko wrote:
> Am Montag, 27. Februar 2017 um 18:28:28, schrieb Pavel Sanda 
> > Christian Ridderström wrote:
> > > Hi,
> > > 
> > > This is a CI job I'm working on that's intended to run automated tests:
> > > 
> > >https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/
> > 
> > How often do you intend to run those?
> > While compile errs tend to get fixed quickly I see quite longer
> > fixing time for automated test (sometimes stuff needs to be discussed etc),
> > so we don't want to shout at mailling list every day about it.

Do you really think 1 email a day would be a bother? For me it would
not, but I understand that this is subjective.

> In contrary, it will show us pretty soon if a commit broke anything.

+1

I think the important thing is to note that the tests are only valid for
an up-to-date TL system. If the texlive packages are used from the
Ubuntu repos and no update is done, then I am sure there would be
failures. A tlmgr update needs to be run, e.g. every week. In fact,
every week or so I would recommend:

1. run the ctests
2. update with tlmgr
3. run the ctests again

That way we could see exactly which tests fail due to the tlmgr update.

Scott


signature.asc
Description: PGP signature

Re: Working on CI job that runs automated tests

[Jean-Pierre Chrétien]

Le 27/02/2017 à 09:48, Jean-Pierre Chrétien a écrit :



I proposed recently to add something about this,  as I made a list of 
dependencies that are not in TL.


Here is the tds of the additional LaTeX stuff which is not un TL2016 and is 
needed to run ctest on unreliable templates and examples files.


http://dl.free.fr/rjsiLvxLE

The agutex.cls package is missing, I could not find it as it is obsoleted by 
agujournal.cls.


Attached is the listing of the files.

--
Jean-Pierre

Archive:  tds_neededForUnreliable.zip
  Length  DateTimeName
-  -- -   
0  2017-02-07 18:08   tex/latex/foiltex/
 3718  2017-02-07 18:03   tex/latex/foiltex/foil20.clo
 2385  2017-02-07 18:03   tex/latex/foiltex/foils.sty
 3208  2017-02-07 18:03   tex/latex/foiltex/foilshrt.clo
 3884  2017-02-07 18:03   tex/latex/foiltex/foil17.clo
11604  2017-02-07 18:03   tex/latex/foiltex/fltfonts.def
 3831  2017-02-07 18:03   tex/latex/foiltex/foil25.clo
 3888  2017-02-07 18:03   tex/latex/foiltex/foil30.clo
27886  2017-02-07 18:03   tex/latex/foiltex/foils.cls
0  2017-02-07 18:39   tex/latex/aa/
65331  2016-09-08 16:09   tex/latex/aa/aa.cls
0  2017-02-08 11:16   tex/latex/knitr/
  810  2013-11-03 21:59   tex/latex/knitr/knitr.sty
 2853  2013-11-03 21:59   tex/latex/knitr/Sweavel.sty
22449  2015-10-19 22:32   tex/latex/knitr/framed.sty
0  2017-02-08 17:46   tex/latex/springer/
0  2017-02-08 17:41   tex/latex/springer/templates/
 1735  2007-10-17 00:00   tex/latex/springer/templates/book.tex
  835  2007-06-29 00:00   tex/latex/springer/templates/acronym.tex
16074  2010-10-20 00:00   tex/latex/springer/templates/author.tex
  512  2007-06-29 00:00   tex/latex/springer/templates/acknow.tex
62739  2007-06-29 00:00   tex/latex/springer/templates/figure.eps
  535  2007-06-29 00:00   tex/latex/springer/templates/part.tex
  502  2007-06-29 00:00   tex/latex/springer/templates/dedic.tex
 8269  2007-06-29 00:00   tex/latex/springer/templates/referenc.tex
 3160  2007-06-29 00:00   tex/latex/springer/templates/appendix.tex
  321  2007-06-29 00:00   tex/latex/springer/templates/solutions.tex
 1307  2007-06-29 00:00   tex/latex/springer/templates/glossary.tex
14338  2007-10-17 00:00   tex/latex/springer/templates/chapter.tex
 1116  2007-06-29 00:00   tex/latex/springer/templates/preface.tex
  834  2007-06-29 00:00   tex/latex/springer/templates/foreword.tex
0  2011-02-22 14:31   tex/latex/springer/deutsch/
 1331  2007-06-29 00:00   tex/latex/springer/deutsch/liesmich.txt
  190  2007-06-29 00:00   tex/latex/springer/deutsch/svindd.ist
31706  2007-06-29 00:00   tex/latex/springer/quickstart.pdf
   118729  2008-06-24 00:00   tex/latex/springer/instruct.pdf
   123690  2008-06-24 00:00   tex/latex/springer/authinst.pdf
50988  2008-10-01 00:00   tex/latex/springer/usrguid3.dvi
   120184  2010-10-20 00:00   tex/latex/springer/authsamp.pdf
   190374  2008-10-01 00:00   tex/latex/springer/usrguid3.pdf
0  2011-03-14 14:48   tex/latex/springer/editor/
0  2011-03-14 14:48   tex/latex/springer/editor/templates/
  835  2007-06-29 00:00   tex/latex/springer/editor/templates/acronym.tex
16108  2010-10-20 00:00   tex/latex/springer/editor/templates/author.tex
  678  2007-06-29 00:00   tex/latex/springer/editor/templates/cblist.tex
  512  2007-06-29 00:00   tex/latex/springer/editor/templates/acknow.tex
62739  2007-06-29 00:00   tex/latex/springer/editor/templates/figure.eps
 1895  2007-06-29 00:00   tex/latex/springer/editor/templates/editor.tex
  535  2007-06-29 00:00   tex/latex/springer/editor/templates/part.tex
  493  2007-06-29 00:00   tex/latex/springer/editor/templates/dedic.tex
 8251  2007-06-29 00:00   tex/latex/springer/editor/templates/referenc.tex
 3181  2007-09-26 00:00   tex/latex/springer/editor/templates/appendix.tex
 1307  2007-06-29 00:00   tex/latex/springer/editor/templates/glossary.tex
 1082  2007-06-29 00:00   tex/latex/springer/editor/templates/preface.tex
  787  2007-06-29 00:00   tex/latex/springer/editor/templates/foreword.tex
   107317  2007-06-29 00:00   tex/latex/springer/editor/edinst.pdf
   126330  2010-10-20 00:00   tex/latex/springer/editor/edsamp.pdf
49110  2010-11-25 10:43   tex/latex/springer/svjour3.cls
 5509  2010-09-16 12:45   tex/latex/springer/template.tex
   173631  2017-02-08 17:17   tex/latex/springer/svjour3.zip
   623954  2011-09-15 11:49   tex/latex/springer/example.pdf
   190989  2007-06-29 00:00   tex/latex/springer/refguide.pdf
 1127  2010-11-25 10:31   tex/latex/springer/history.txt
28600  2004-11-08 11:50   tex/latex/springer/spphys.bst
0  2017-02-08 17:41   tex/latex/springer/styles/
33250  2007-06-29 00:00   tex/latex/springer/styles/spbasic.bst
  180  2007-06-29 00:00   tex/latex/springer/styles/svind.ist
68383  

Re: Working on CI job that runs automated tests

[Kornel Benko]

Am Montag, 27. Februar 2017 um 18:28:28, schrieb Pavel Sanda 
> Christian Ridderström wrote:
> > Hi,
> > 
> > This is a CI job I'm working on that's intended to run automated tests:
> > 
> >https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/
> 
> How often do you intend to run those?
> While compile errs tend to get fixed quickly I see quite longer
> fixing time for automated test (sometimes stuff needs to be discussed etc),
> so we don't want to shout at mailling list every day about it.
> 
> Pavel

In contrary, it will show us pretty soon if a commit broke anything.

-1

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Working on CI job that runs automated tests

[Pavel Sanda]

Christian Ridderström wrote:
> Hi,
> 
> This is a CI job I'm working on that's intended to run automated tests:
> 
>https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/

How often do you intend to run those?
While compile errs tend to get fixed quickly I see quite longer
fixing time for automated test (sometimes stuff needs to be discussed etc),
so we don't want to shout at mailling list every day about it.

Pavel

Re: Working on CI job that runs automated tests

[Kornel Benko]

Am Montag, 27. Februar 2017 um 22:30:16, schrieb Christian Ridderström 

> On 27 February 2017 at 17:34, Jean-Pierre Chrétien <
> jeanpierre.chret...@free.fr> wrote:
> 
> > I proposed recently to add something about this,  as I made a list of
> >> dependencies that are not in TL.
> >>
> >
> > In fact, you may check the thread beginning with this message:
> >
> > https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg198687.html
> 
> 
> I quickly read through the thread, but I'll have to read it more carefully.
> 
> However, instead of having lots of failed tests (2000), perhaps we can
> start with making a smaller set of tests that all pass?
> 
> We can easily create several CI jobs, where each CI job executes a
> different set of tests.
> This would also make it less tedious to wait for a CI job to complete.
> And if two CI workers isn't enough (speed wise), we could add more,
> allowing several CI test jobs to run in parallel.

That is not possible. But you can run tests like 'ctest -j6'.

> So what would be a good set of tests to start with?
> Preferably a set of tests that require some relatively easy dependencies
> like LiveTex and ImageMagick and perhaps some other things.

I vote for files in the doc dir, where should all pass.
# ctest -R '/doc/.*pdf2' => 94 tests here

> Then I can also go about adding that CI job to the regular list of jobs
> that are run, and have the CI job e-mail the devel list when it fails.
> /Christian

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: Working on CI job that runs automated tests

[Christian Ridderström]

On 27 February 2017 at 17:34, Jean-Pierre Chrétien <
jeanpierre.chret...@free.fr> wrote:

> I proposed recently to add something about this,  as I made a list of
>> dependencies that are not in TL.
>>
>
> In fact, you may check the thread beginning with this message:
>
> https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg198687.html


I quickly read through the thread, but I'll have to read it more carefully.

However, instead of having lots of failed tests (2000), perhaps we can
start with making a smaller set of tests that all pass?

We can easily create several CI jobs, where each CI job executes a
different set of tests.
This would also make it less tedious to wait for a CI job to complete.
And if two CI workers isn't enough (speed wise), we could add more,
allowing several CI test jobs to run in parallel.

So what would be a good set of tests to start with?
Preferably a set of tests that require some relatively easy dependencies
like LiveTex and ImageMagick and perhaps some other things.
Then I can also go about adding that CI job to the regular list of jobs
that are run, and have the CI job e-mail the devel list when it fails.
/Christian

Re: Working on CI job that runs automated tests

[Christian Ridderström]

Apologies, accidentally sent only to Kornel rather than the list, reposting
now.

On 27 February 2017 at 09:18, Kornel Benko  wrote:

> > https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/
> lastCompletedBuild/testReport/
>
> Looking inside this I see for instance in the first entry
> UNRELIABLE.WRONG_OUTPUT_export/doc/es/EmbeddedObjects_dvi3_texF:
> ...
> support/Systemcall.cpp (279): Systemcall: 'git ls-files
> "EmbeddedObjects_dvi3_texF.lyx"' did not start!
> ...
> sh: 1: convert: not found
> ...
> Looks like imagemagic is not installed
> So I don't wonder too much.
>

That's just the kind of feedback I need.

FYI, I've now:
a) Added 'imagemagick' to things to install in the script
'install_dependencies.sh'
b) Committed and pushed the change to GitHub
c) DockerHub has now detected the change in the repository and is
automatically rebuilding the Docker image
This will take some minutes at least - TexLive isn't light weight :-)
Build status is visible here:
https://hub.docker.com/r/lyxproject/build-lyx-using-
ubuntu-latest-qt5-ctest/builds/
d) Meanwhile I've deleted the Docker image from the CI, so when next
someone/something triggers
the CI job, the CI node will automatically pull down the Docker image
from Docker hub.

As the CI job checks the repo every 8 hours, it'll run pretty soon. I
checked and it'll run by midnight.
So tomorrow morning we can see how much adding imagemagick helped.
/Christian
Update: The CI job is running, it's been going for two hours but will
likely need at least 4.5 hours.

Re: Working on CI job that runs automated tests

[Jean-Pierre Chrétien]

Le 27/02/2017 à 09:48, Jean-Pierre Chrétien a écrit :

Le 27 février 2017 08:56:50 CET, "Christian Ridderström"  a écrit 
:

Hi,

This is a CI job I'm working on that's intended to run automated tests:

  https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/

Currently there's about 2300 failing tests (out of 6000), so I'm
guessing
more stuff needs to be installed.

Detailed results via this link:

https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/lastCompletedBuild/testReport/
and here's a link to the workspace with the "build/Testing/"-folder
https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/ws/build/Testing/

The environment is Ubuntu + qt5 + CMake, and I've done
  apt-get install texlive-full

Not sure how I could proceed now, I think I need some help in figuring
out
how to get more of the tests to pass. Perhaps I need to install more
dependencies?  Perhaps the Docker image needs to be running XWindow?
Note: I have looked in Developer.lyx but didn't see these kinds of
details,


I proposed recently to add something about this,  as I made a list of 
dependencies that are not in TL.


In fact, you may check the thread beginning with this message:

https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg198687.html

--
Jean-Pierre

Re: Working on CI job that runs automated tests

[Jean-Pierre Chrétien]

Le 27 février 2017 08:56:50 CET, "Christian Ridderström"  a écrit 
:
>Hi,
>
>This is a CI job I'm working on that's intended to run automated tests:
>
>   https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/
>
>Currently there's about 2300 failing tests (out of 6000), so I'm
>guessing
>more stuff needs to be installed.
>
>Detailed results via this link:
>
>https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/lastCompletedBuild/testReport/
>and here's a link to the workspace with the "build/Testing/"-folder
>https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/ws/build/Testing/
>
>The environment is Ubuntu + qt5 + CMake, and I've done
>   apt-get install texlive-full
>
>Not sure how I could proceed now, I think I need some help in figuring
>out
>how to get more of the tests to pass. Perhaps I need to install more
>dependencies?  Perhaps the Docker image needs to be running XWindow?
>Note: I have looked in Developer.lyx but didn't see these kinds of
>details,

I proposed recently to add something about this,  as I made a list of 
dependencies that are not in TL.

I'll keep you posted when I'm back home.



-- 
Jean-Pierre

Re: Working on CI job that runs automated tests

[Kornel Benko]

Am Montag, 27. Februar 2017 um 08:56:50, schrieb Christian Ridderström 

> Hi,
> 
> This is a CI job I'm working on that's intended to run automated tests:
> 
>https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/
> 
> Currently there's about 2300 failing tests (out of 6000), so I'm guessing
> more stuff needs to be installed.
> 
> Detailed results via this link:
> 
> https://ci.inria.fr/lyx/job/testing-Jenkins/job/ctest/lastCompletedBuild/testReport/

Looking inside this I see for instance in the first entry 
UNRELIABLE.WRONG_OUTPUT_export/doc/es/EmbeddedObjects_dvi3_texF:
...
support/Systemcall.cpp (279): Systemcall: 'git ls-files 
"EmbeddedObjects_dvi3_texF.lyx"' did not start!
...
sh: 1: convert: not found
...
Looks like imagemagic is not installed
So I don't wonder too much.

Kornel

P.S.
Nice errors protocol BTW.

signature.asc
Description: This is a digitally signed message part.