Re: [cmake-developers] CTest custom HTML

[Charles Huet]

Yeah, makes sense.

Thanks so much for your time

Le lun. 14 mars 2016 à 15:58, Brad King  a écrit :

> On 03/14/2016 05:13 AM, Charles Huet wrote:
> > If the CTest parser parsed the output as XML and not simple text, it
> should be
> > able to recognize CDATA from actual markup.
> [snip]
> > If you don't think this is doable (or you don't want to integrate this
> kind
> > of behavior into CTest) we'll customize CDash and I'll stop badgering
> you about this.
> >
> > But I'd rather do this as a last resort.
>
> I'm sorry but I don't want to introduce any special interpretation of test
> output beyond the hooks that are already present.  In principle the test
> output is just plain text and the fact that it could be interpreted as
> XML or JSON or some other markup is not CTest's business.  It is encoded
> in Test.xml for submission to CDash or some other similar system with
> all escaping needed to communicate the real test output to that system.
>
> -Brad
>
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest custom HTML

[Brad King]

On 03/14/2016 05:13 AM, Charles Huet wrote:
> If the CTest parser parsed the output as XML and not simple text, it should be
> able to recognize CDATA from actual markup.
[snip]
> If you don't think this is doable (or you don't want to integrate this kind
> of behavior into CTest) we'll customize CDash and I'll stop badgering you 
> about this.
> 
> But I'd rather do this as a last resort.

I'm sorry but I don't want to introduce any special interpretation of test
output beyond the hooks that are already present.  In principle the test
output is just plain text and the fact that it could be interpreted as
XML or JSON or some other markup is not CTest's business.  It is encoded
in Test.xml for submission to CDash or some other similar system with
all escaping needed to communicate the real test output to that system.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest custom HTML

[Charles Huet]

>You could use DartMeasurementFile to upload arbitrary
files with the CTest submission, and then teach CDash what to do with
them.

Hmm, this sounds to me like too much custom logic would be in CDash, which
would never make it upstream, and we have bad experiences maintaining this
kind of patch sets.

>I'd prefer not to add yet another dependency (on xslt).
Yeah, I guessed this would be your answer, it makes sense.

I have had another, simpler (?) idea that would fit everybody's
requirements.
If the CTest parser parsed the output as XML and not simple text, it should
be able to recognize CDATA from actual markup.
Thus replacing the special chars could be done when reading, and the
cmCtestTestHandler::GenerateDartOutput() function could be adapted
easily(?) to write the read XML.

I'm not sure how feasible this would be, but this sounds like the best way
to integrate custom XML in a CTest report.

If you don't think this is doable (or you don't want to integrate this kind
of behavior into CTest) we'll customize CDash and I'll stop badgering you
about this.

But I'd rather do this as a last resort.

Thanks


Le jeu. 10 mars 2016 à 15:29, Brad King  a écrit :

> On 03/10/2016 03:29 AM, Charles Huet wrote:
> > Now the big difficulty in this would be to integrate XSLT2 capabilities
> into CMake.
> [snip]
> > Do you think this coule be viable ?
>
> I'd prefer not to add yet another dependency (on xslt).  CTest defines
> nothing about the format of the test output, besides DartMeasurement hooks.
> It isn't guaranteed to even be displayed on a web page where having
> custom HTML even makes sense.  This really should be done on the
> CDash side.  You could use DartMeasurementFile to upload arbitrary
> files with the CTest submission, and then teach CDash what to do with
> them.
>
> -Brad
>
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest custom HTML

[Brad King]

On 03/10/2016 03:29 AM, Charles Huet wrote:
> Now the big difficulty in this would be to integrate XSLT2 capabilities into 
> CMake.
[snip]
> Do you think this coule be viable ?

I'd prefer not to add yet another dependency (on xslt).  CTest defines
nothing about the format of the test output, besides DartMeasurement hooks.
It isn't guaranteed to even be displayed on a web page where having
custom HTML even makes sense.  This really should be done on the
CDash side.  You could use DartMeasurementFile to upload arbitrary
files with the CTest submission, and then teach CDash what to do with
them.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest custom HTML

[Charles Huet]

Hi,

this is very interesting, but it does not fully solve the problem of not
being able to have some custom HTML sent to CDash directly.

I perused through the code to try and find a solution, and I think I have
an idea.
>From what I understand, the 'DartMeasurement' tokens are replaced with some
logic into  and such.

If this were to happen in an XSLT file, that could be specified to CTest,
then not only would the current behavior for vanilla CTest/CDash remain the
same, it would be easier to maintain and make evolve as it would not
require recompilation.
On top of that, any custom behavior like ours is very easy to support, just
specify a customn XSLT sheet, and have CTest use this one instead of the
default.

Using an XSLT2  character-map would allow for the current behavior of
replacing characters like '>', '<' and other HTML-breaking things quite
easily.

Now the big difficulty in this would be to integrate XSLT2 capabilities
into CMake.
As far as I understand, CMake tries very hard to be as cross-platform as
possible in a single package, which means no hard dependencies, and any
third-party lib gets integrated into the CMake source tree (such as
cmbzip2, cmcurl, etc).

I only know of libxslt, and have no idea if it can be integrated in this
way, or how much work that would be.
Maybe there are better, lighter alternatives out there, but I am not
knowledgable in xslt libraries.

Do you think this coule be viable ?


Le mer. 9 mars 2016 à 14:36, Brad King  a écrit :

> On 03/09/2016 04:13 AM, Charles Huet wrote:
> > We used to use this, by creating a composite image containing the
> reference,
> > generated and diff, but in some cases comparing accurately was tedious.
> > Having the 3 images separately allows us to open each in a tab and
> quickly
> > switch from one to the other, making human comparison much easier.
> >
> > Also, we have added some HTML buttons and tables, which we would lose by
> using this.
>
> This work seems more suited for development on the CDash side to influence
> how it displays the images.
>
> For VTK testing we have an interactive slider to help compare the images:
>
>  https://open.cdash.org/testDetails.php?test=422454265=4246600
>
> -Brad
>
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest custom HTML

[Brad King]

On 03/09/2016 04:13 AM, Charles Huet wrote:
> We used to use this, by creating a composite image containing the reference,
> generated and diff, but in some cases comparing accurately was tedious.
> Having the 3 images separately allows us to open each in a tab and quickly
> switch from one to the other, making human comparison much easier.
> 
> Also, we have added some HTML buttons and tables, which we would lose by 
> using this.

This work seems more suited for development on the CDash side to influence
how it displays the images.

For VTK testing we have an interactive slider to help compare the images:

 https://open.cdash.org/testDetails.php?test=422454265=4246600

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest custom HTML

[Charles Huet]

Sadly this is quite not good enough.
We used to use this, by creating a composite image containing the
reference, generated and diff, but in some cases comparing accurately was
tedious.
Having the 3 images separately allows us to open each in a tab and quickly
switch from one to the other, making human comparison much easier.

Also, we have added some HTML buttons and tables, which we would lose by
using this.



Le mar. 8 mars 2016 à 19:27, Brad King  a écrit :

> On 03/08/2016 11:09 AM, Charles Huet wrote:
> > mostly useful to display images, since some of our tests compare
> > images with ImageMagick
>
> CTest/CDash already supports sending images for display.  It was
> developed for the same purpose for VTK testing.  The test output
> can contain
>
>type="image/png">/path/to/file.png
>
> CTest will load the named file and send it as part of the submission
> to CDash.
>
> -Brad
>
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CTest custom HTML

[Brad King]

On 03/08/2016 11:09 AM, Charles Huet wrote:
> mostly useful to display images, since some of our tests compare
> images with ImageMagick

CTest/CDash already supports sending images for display.  It was
developed for the same purpose for VTK testing.  The test output
can contain

  /path/to/file.png

CTest will load the named file and send it as part of the submission
to CDash.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] CTest custom HTML

[Charles Huet]

Hi,

to customize our test results, we developped a small test wrapper that
outputs some HTML markup (mostly useful to display images, since some of
our tests compare images with ImageMagick).
However, this does not work since CMake 3.3 because of the refactoring of
ctest to use cmXMLWriter, because cmXMLWriter is smart and wants to output
correct XML, and thus escapes everything.

While I completely understand and applaud the motivation and work behind
cmXMLWriter, it annoys me that we cannot have our custom reporting anymore.

Sadly, I have no idea how to have both the correctness provided by
cmXMLWriter and the flexibility of outputting custom HTML via the test's
stdout.

How could this be done with CMake > 3.3 ?
If it needs additional development on the CTest side I would of course try
and accomplish it, if anyone has an idea.

Thanks
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers