Re: ctests: give output of LaTeX error

[Scott Kostyshak]

On Tue, Oct 11, 2016 at 01:51:05PM +0200, Kornel Benko wrote:

> Try this patch. Setting env var LYX_DEBUG_LATEX to '1' will produce logs with 
> latex.
> No change of logs if set to '0' or not set at all.

Looks good. If that covers Günter's request I think it makes sense to go
in, at least until one of the tickets is fixed that allows us to view
only the most recent LaTeX error.

Scott


signature.asc
Description: PGP signature

Re: ctests: give output of LaTeX error

[Kornel Benko]

Am Montag, 10. Oktober 2016 um 19:44:09, schrieb Kornel Benko 
> Am Montag, 10. Oktober 2016 um 09:56:00, schrieb Scott Kostyshak 
> 
> > On Mon, Oct 10, 2016 at 07:56:39AM +, Guenter Milde wrote:
> >
> > > > Difficult.
> > >
> > > Then the debug output for inverted and suspended tests would be fine as
> > > workaround until we have the "keep the tempdir" option.
> >
> > +1
> >
> > Also note that it is easy to turn the -dbg on. For example, you might
> > want to do something like this:
> >
> > 1. run all ctests
> > 2. apply local patch that uses "-dbg latex" for all tests (basically
> >just remove the "if" in Kornel's current patch)
> > 3. ctest --rerun-failed
> > 4. revert local patch that turns debugging on (just to make sure it is
> >not accidentally committed).
> >
> > If it would help, I could make a script that does the above.
>
> Since ctest runs our cmake scripts, it is also possible to depend on 
> environment.
> I am reluctant though, making test results even more depending.
>
> > Scott

Try this patch. Setting env var LYX_DEBUG_LATEX to '1' will produce logs with 
latex.
No change of logs if set to '0' or not set at all.

Korneldiff --git a/development/autotests/export.cmake b/development/autotests/export.cmake
index 09e7ec7..fea5e25 100755
--- a/development/autotests/export.cmake
+++ b/development/autotests/export.cmake
@@ -126,10 +126,15 @@ if (extension MATCHES "\\.lyx$")
 set(LYX_SOURCE ${result_file_name})
   endforeach()
 else()
-  message(STATUS "Executing ${lyx} -userdir \"${LYX_TESTS_USERDIR}\" -E ${format} ${result_file_name} \"${LYX_SOURCE}\"")
+  if ($ENV{LYX_DEBUG_LATEX})
+set(LatexDebugParam -dbg latex)
+  else()
+set(LatexDebugParam)
+  endif()
+  message(STATUS "Executing ${lyx} ${LatexDebugParam} -userdir \"${LYX_TESTS_USERDIR}\" -E ${format} ${result_file_name} \"${LYX_SOURCE}\"")
   file(REMOVE ${result_file_name})
   execute_process(
-COMMAND ${lyx} -userdir "${LYX_TESTS_USERDIR}" -E ${format} ${result_file_name} "${LYX_SOURCE}"
+COMMAND ${lyx} ${LatexDebugParam} -userdir "${LYX_TESTS_USERDIR}" -E ${format} ${result_file_name} "${LYX_SOURCE}"
 RESULT_VARIABLE _err)

   #check if result file created


signature.asc
Description: This is a digitally signed message part.

Re: ctests: give output of LaTeX error

[Kornel Benko]

Am Montag, 10. Oktober 2016 um 09:56:00, schrieb Scott Kostyshak 

> On Mon, Oct 10, 2016 at 07:56:39AM +, Guenter Milde wrote:
> 
> > > Difficult. 
> > 
> > Then the debug output for inverted and suspended tests would be fine as
> > workaround until we have the "keep the tempdir" option.
> 
> +1
> 
> Also note that it is easy to turn the -dbg on. For example, you might
> want to do something like this:
> 
> 1. run all ctests
> 2. apply local patch that uses "-dbg latex" for all tests (basically
>just remove the "if" in Kornel's current patch)
> 3. ctest --rerun-failed
> 4. revert local patch that turns debugging on (just to make sure it is
>not accidentally committed).
> 
> If it would help, I could make a script that does the above.

Since ctest runs our cmake scripts, it is also possible to depend on 
environment.
I am reluctant though, making test results even more depending.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: ctests: give output of LaTeX error

[Scott Kostyshak]

On Mon, Oct 10, 2016 at 07:56:39AM +, Guenter Milde wrote:

> > Difficult. 
> 
> Then the debug output for inverted and suspended tests would be fine as
> workaround until we have the "keep the tempdir" option.

+1

Also note that it is easy to turn the -dbg on. For example, you might
want to do something like this:

1. run all ctests
2. apply local patch that uses "-dbg latex" for all tests (basically
   just remove the "if" in Kornel's current patch)
3. ctest --rerun-failed
4. revert local patch that turns debugging on (just to make sure it is
   not accidentally committed).

If it would help, I could make a script that does the above.

Scott


signature.asc
Description: PGP signature

Re: ctests: give output of LaTeX error

[Guenter Milde]

Dear Kornel and Scott,

On 2016-10-09, Kornel Benko wrote:
> Am Sonntag, 9. Oktober 2016 um 17:55:49, schrieb Scott Kostyshak 
> 
>> On Sun, Oct 09, 2016 at 09:28:47PM +, Guenter Milde wrote:
>> > On 2016-10-09, Scott Kostyshak wrote:

...
>> > > ... Günter, do you want the log output for
>> > > the non-inverted tests also?
>> > 
>> > Preferabely yes, because then I know if inversion is required if a test 
>> > fails.

...

>> >  
>> > Maybe the dbg could be collected and only appended to the log if there was
>> > an error...

> Difficult. 

Then the debug output for inverted and suspended tests would be fine as
workaround until we have the "keep the tempdir" option.

With this, we could append the log (or an excerpt) in case of error for
(failure of normal tests and no failure for inverted tests).

Günter

Re: ctests: give output of LaTeX error

[Kornel Benko]

Am Sonntag, 9. Oktober 2016 um 17:55:49, schrieb Scott Kostyshak 

> On Sun, Oct 09, 2016 at 09:28:47PM +, Guenter Milde wrote:
> > On 2016-10-09, Scott Kostyshak wrote:
> 
> > > Good idea. Assuming we do not have many inverted tests that would keep
> > > the file size of the log down. Günter, do you want the log output for
> > > the non-inverted tests also?
> > 
> > Preferabely yes, because then I know if inversion is required if a test 
> > fails.
> > Otherwise, a new failure would require 
> >  * inversion (as TODO)
> >  * rerunning cmake
> >  * inspection of the log
> >  * sort of the inversion-pattern or fix of the installation or ignoring ...
> >  * rerunning cmake
> 
> Makes sense.
> 
> >  
> > Maybe the dbg could be collected and only appended to the log if there was
> > an error...
> 
> I'm not sure this is possible. Maybe Kornel is aware of some magic to
> make it happen though.

Difficult. We would
1.) call lyx with '-dbg latex'
2.) collect output in cmake variable
3a.) In case of error, output the the log
3b.) else strip the log of latex messages.
4.) print the stripped output
Unfortunately the lyx-latex debug messages are not easy to filter

Normally they start with
SomeFile.cpp (xxx): Log line: some latex log
but if the log is multi-line, we are lost.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.

Re: ctests: give output of LaTeX error

[Scott Kostyshak]

On Sun, Oct 09, 2016 at 09:28:47PM +, Guenter Milde wrote:
> On 2016-10-09, Scott Kostyshak wrote:

> > Good idea. Assuming we do not have many inverted tests that would keep
> > the file size of the log down. Günter, do you want the log output for
> > the non-inverted tests also?
> 
> Preferabely yes, because then I know if inversion is required if a test fails.
> Otherwise, a new failure would require 
>  * inversion (as TODO)
>  * rerunning cmake
>  * inspection of the log
>  * sort of the inversion-pattern or fix of the installation or ignoring ...
>  * rerunning cmake

Makes sense.

>  
> Maybe the dbg could be collected and only appended to the log if there was
> an error...

I'm not sure this is possible. Maybe Kornel is aware of some magic to
make it happen though.

Scott


signature.asc
Description: PGP signature

Re: ctests: give output of LaTeX error

[Guenter Milde]

On 2016-10-09, Scott Kostyshak wrote:

> [-- Type: text/plain, Encoding: quoted-printable --]

> On Sun, Oct 09, 2016 at 07:46:24PM +0200, Kornel Benko wrote:
>> Am Sonntag, 9. Oktober 2016 um 19:21:57, schrieb Kornel Benko 
>> 
>> > Am Sonntag, 9. Oktober 2016 um 12:38:44, schrieb Scott Kostyshak 
>> > 

>> > > Is it possible to show the -dbg output only when there was an error? I
>> > > don't know if our ctest framework allows us to do that.
>> > 
>> > It is possible, but we have to know that there is an error prior to ctest 
>> > call.
>> > 

>> Like attached?

> Good idea. Assuming we do not have many inverted tests that would keep
> the file size of the log down. Günter, do you want the log output for
> the non-inverted tests also?

Preferabely yes, because then I know if inversion is required if a test fails.
Otherwise, a new failure would require 
 * inversion (as TODO)
 * rerunning cmake
 * inspection of the log
 * sort of the inversion-pattern or fix of the installation or ignoring ...
 * rerunning cmake
 
Maybe the dbg could be collected and only appended to the log if there was
an error...
 
Günter

Re: ctests: give output of LaTeX error

[Scott Kostyshak]

On Sun, Oct 09, 2016 at 07:46:24PM +0200, Kornel Benko wrote:
> Am Sonntag, 9. Oktober 2016 um 19:21:57, schrieb Kornel Benko 
> > Am Sonntag, 9. Oktober 2016 um 12:38:44, schrieb Scott Kostyshak 
> > 

> > > Is it possible to show the -dbg output only when there was an error? I
> > > don't know if our ctest framework allows us to do that.
> > 
> > It is possible, but we have to know that there is an error prior to ctest 
> > call.
> > 
> 
> Like attached?

Good idea. Assuming we do not have many inverted tests that would keep
the file size of the log down. Günter, do you want the log output for
the non-inverted tests also?

Scott


signature.asc
Description: PGP signature

Re: ctests: give output of LaTeX error

[Kornel Benko]

Am Sonntag, 9. Oktober 2016 um 19:21:57, schrieb Kornel Benko 
> Am Sonntag, 9. Oktober 2016 um 12:38:44, schrieb Scott Kostyshak 
> 
> > When encountering a ctest failure, we must run the test manually to see
> > what the LaTeX error. It would be nice to be able to see the LaTeX error
> > from the ctest logs. The following tickets are related:
> > http://www.lyx.org/trac/ticket/9931
> > http://www.lyx.org/trac/ticket/9866
> >
> > One possibility would be for the ctests to call LyX with
> >
> >   -dbg latex
>
> Yes, will do.
>
> > which does indeed show the error.
> >
> > I don't think this is a good idea though, because The LastTest.log file
> > already has 1653656 lines and is 110M. With this command, I think the
> > file will get much larger.
> >
> > Is it possible to show the -dbg output only when there was an error? I
> > don't know if our ctest framework allows us to do that.
>
> It is possible, but we have to know that there is an error prior to ctest 
> call.
>

Like attached?

Kornel
diff --git a/development/autotests/export.cmake b/development/autotests/export.cmake
index 09e7ec7..6c02750 100755
--- a/development/autotests/export.cmake
+++ b/development/autotests/export.cmake
@@ -126,10 +126,15 @@ if (extension MATCHES "\\.lyx$")
 set(LYX_SOURCE ${result_file_name})
   endforeach()
 else()
-  message(STATUS "Executing ${lyx} -userdir \"${LYX_TESTS_USERDIR}\" -E ${format} ${result_file_name} \"${LYX_SOURCE}\"")
+  if (inverted)
+set(LatexDebugParam "-dbg latex")
+  else()
+set(LatexDebugParam "")
+  endif()
+  message(STATUS "Executing ${lyx} ${LatexDebugParam} -userdir \"${LYX_TESTS_USERDIR}\" -E ${format} ${result_file_name} \"${LYX_SOURCE}\"")
   file(REMOVE ${result_file_name})
   execute_process(
-COMMAND ${lyx} -userdir "${LYX_TESTS_USERDIR}" -E ${format} ${result_file_name} "${LYX_SOURCE}"
+COMMAND ${lyx} ${LatexDebugParam} -userdir "${LYX_TESTS_USERDIR}" -E ${format} ${result_file_name} "${LYX_SOURCE}"
 RESULT_VARIABLE _err)

   #check if result file created


signature.asc
Description: This is a digitally signed message part.

Re: ctests: give output of LaTeX error

[Kornel Benko]

Am Sonntag, 9. Oktober 2016 um 12:38:44, schrieb Scott Kostyshak 

> When encountering a ctest failure, we must run the test manually to see
> what the LaTeX error. It would be nice to be able to see the LaTeX error
> from the ctest logs. The following tickets are related:
> http://www.lyx.org/trac/ticket/9931
> http://www.lyx.org/trac/ticket/9866
> 
> One possibility would be for the ctests to call LyX with 
> 
>   -dbg latex

Yes, will do.

> which does indeed show the error.
> 
> I don't think this is a good idea though, because The LastTest.log file
> already has 1653656 lines and is 110M. With this command, I think the
> file will get much larger.
> 
> Is it possible to show the -dbg output only when there was an error? I
> don't know if our ctest framework allows us to do that.

It is possible, but we have to know that there is an error prior to ctest call.

> Scott

Kornel

signature.asc
Description: This is a digitally signed message part.