Hi,
attached are two patches for ctest and cdash from Volkan (on CC).
They are not ready for inclusion, but before we continue to work in this, we'd
like to get some comments whether this is a good idea, the right approach,
etc.
So, here we go.
We are using cdash a lot, and for many of our tests the log is quite verbose
and somewhat hard to read. When looking for the reason why a test failed, it
can be somewhat hard to find the line which contains "ERROR".
Also, a lot of stuff is repeated every line, which also makes the log harder
to read, e.g. the directory part of the file where the log came from:
<timestamp> /some/where/deep/in/a/subdir/test.py: Something...
<timestamp> /some/where/deep/in/a/subdir/test.py: happend...
<timestamp> /some/where/deep/in/a/subdir/test.py: here...
<timestamp> /some/where/deep/in/a/subdir/test.py: ...
The directory part is actually so wide that we have to scroll to see actual
text.
So, to help with this, we came up with the idea to use HTML to format the log
so that it becomes easier to read.
The attached patch for ctest adds a new test property GENERATE_HTML, and if
this is enabled, ctest generates HTML. It searches the lines which made the
test fail and colors them red, and the lines which make a test pass are
colored green.
This makes it easier to find the fail/succeed lines.
(In the patch the fail strings are still hardcoded, this would have to use the
respective test property.)
On the cdash side, if "</span>" is found in the log, it is considered HTML and
not escaped. I guess this should be done in a different way, which does not
involve parsing the log text.
Comments so far ?
For making our log lines shorter, we are not sure yet how to do that.
We could add support for squish XML to ctest, so that ctest reads the squish
XML and converts it to HTML which is in some way shorter, maybe a HTML table,
or something.
Or we could add a test property with a regular expression, if that regexp
matches, ctest replaces it with a shorter version or something.
Or we could modify our test scripts so that they already generate the pretty
HTML, which ctest then simply sends to cdash.
That's the ideas we have so far, but it doesn't feel quite right yet.
So, what do you think ?
Alex
From 446c27d8ff38704ffd6f8c828d49d4e7110fb1ac Mon Sep 17 00:00:00 2001
From: Kitware Robot <[email protected]>
Date: Thu, 4 Apr 2013 00:01:08 -0400
Subject: added html generation capability to colorize plain output
---
Source/CTest/cmCTestRunTest.cxx | 47 +++++++++++++++++++++++++++++++++-
Source/CTest/cmCTestRunTest.h | 7 ++++-
Source/CTest/cmCTestTestHandler.cxx | 5 ++++
Source/CTest/cmCTestTestHandler.h | 1 +
Source/cmSetTestsPropertiesCommand.h | 4 ++-
Source/cmTest.cxx | 4 +++
7 files changed, 66 insertions(+), 4 deletions(-)
diff --git a/Source/CTest/cmCTestRunTest.cxx b/Source/CTest/cmCTestRunTest.cxx
index 5eabf3f..b7cb482 100644
--- a/Source/CTest/cmCTestRunTest.cxx
+++ b/Source/CTest/cmCTestRunTest.cxx
@@ -60,7 +60,17 @@ bool cmCTestRunTest::CheckOutput()
// Store this line of output.
cmCTestLog(this->CTest, HANDLER_VERBOSE_OUTPUT,
this->GetIndex() << ": " << line << std::endl);
- this->ProcessOutput += line;
+
+ if( this->TestProperties->GenerateHtml == true )
+ {
+ // If user set GENERATE_HTML flag
+ this->ProcessOutput += this->StrictCheckColorize(line);
+ }
+ else
+ {
+ this->ProcessOutput += line;
+ }
+
this->ProcessOutput += "\n";
}
else // if(p == cmsysProcess_Pipe_Timeout)
@@ -71,6 +81,41 @@ bool cmCTestRunTest::CheckOutput()
return true;
}
+// text: String to be checked and colorized for PASS and FAIL conditions
+std::string cmCTestRunTest::StrictCheckColorize(const std::string& text)
+{
+ // Check for the following FAIL and PASS patterns and return
+ // the line itself after colorization
+ // If no expressions are found, do nothing
+
+ // NOTE: Braced initializers are not available in < c++11
+
+ const std::string FAILS[] = {"FAIL", "Error", "ERROR", "Fail", "fail", "Fatal", "FATAL"};
+ const std::string PASSES[] = {"Pass", "PASS", "pass"};
+ const std::vector<std::string> fails(FAILS, FAILS + sizeof(FAILS)/sizeof(*FAILS));
+ const std::vector<std::string> passes(PASSES, PASSES + sizeof(PASSES)/sizeof(*PASSES));
+ double cannotFound = std::string::npos;
+ double foundPos = cannotFound;
+
+ for ( int i = 0; i < fails.size(); i++ )
+ {
+ foundPos = text.find(fails[i]);
+ if ( foundPos != cannotFound )
+ {
+ return std::string("<span style='color:#AA0000'>") + text + "</span>";
+ }
+ }
+ for ( int i = 0; i < passes.size(); i++ )
+ {
+ foundPos = text.find(passes[i]);
+ if ( foundPos != cannotFound )
+ {
+ return std::string("<span style='color:#00AA00'>") + text + "</span>";
+ }
+ }
+ return text;
+}
+
//---------------------------------------------------------
// Streamed compression of test output. The compressed data
// is appended to this->CompressedOutput
diff --git a/Source/CTest/cmCTestRunTest.h b/Source/CTest/cmCTestRunTest.h
index 476f3e1..d0312af 100644
--- a/Source/CTest/cmCTestRunTest.h
+++ b/Source/CTest/cmCTestRunTest.h
@@ -49,7 +49,12 @@ public:
// Compresses the output, writing to CompressedOutput
void CompressOutput();
-
+
+ // Do a strict check for fixed FAIL and PASS conditions and colorize the
+ // text if they are found.
+ // text: String to be checked and colorized for PASS and FAIL conditions
+ std::string StrictCheckColorize(const std::string& text);
+
//launch the test process, return whether it started correctly
bool StartTest(size_t total);
//capture and report the test results
diff --git a/Source/CTest/cmCTestTestHandler.cxx b/Source/CTest/cmCTestTestHandler.cxx
index e7491bb..842c8aa 100644
--- a/Source/CTest/cmCTestTestHandler.cxx
+++ b/Source/CTest/cmCTestTestHandler.cxx
@@ -2175,6 +2175,10 @@ bool cmCTestTestHandler::SetTestsProperties(
{
rtit->Directory = val;
}
+ if ( key == "GENERATE_HTML" )
+ {
+ rtit->GenerateHtml = cmSystemTools::IsOn(val.c_str());
+ }
}
}
}
@@ -2250,6 +2254,7 @@ bool cmCTestTestHandler::AddTest(const std::vector<std::string>& args)
test.Cost = 0;
test.Processors = 1;
test.PreviousRuns = 0;
+ test.GenerateHtml = false; // Plain output by default
if (this->UseIncludeRegExpFlag &&
!this->IncludeTestsRegularExpression.find(testname.c_str()))
{
diff --git a/Source/CTest/cmCTestTestHandler.h b/Source/CTest/cmCTestTestHandler.h
index 8e59e59..840f3e1 100644
--- a/Source/CTest/cmCTestTestHandler.h
+++ b/Source/CTest/cmCTestTestHandler.h
@@ -100,6 +100,7 @@ public:
bool RunSerial;
double Timeout;
bool ExplicitTimeout;
+ bool GenerateHtml;
int Index;
//Requested number of process slots
int Processors;
diff --git a/Source/cmSetTestsPropertiesCommand.h b/Source/cmSetTestsPropertiesCommand.h
index 3a59218..421d76a 100644
--- a/Source/cmSetTestsPropertiesCommand.h
+++ b/Source/cmSetTestsPropertiesCommand.h
@@ -65,7 +65,9 @@ public:
"Both PASS_REGULAR_EXPRESSION and FAIL_REGULAR_EXPRESSION expect a "
"list of regular expressions.\n"
"TIMEOUT: Setting this will limit the test runtime to the number of "
- "seconds specified.\n";
+ "seconds specified.\n"
+ "GENERATE_HTML: If set, the test output will be edited to colorize "
+ "PASSED and FAILED tests.\n";
}
cmTypeMacro(cmSetTestsPropertiesCommand, cmCommand);
diff --git a/Source/cmTest.cxx b/Source/cmTest.cxx
index 912ec76..7161001 100644
--- a/Source/cmTest.cxx
+++ b/Source/cmTest.cxx
@@ -202,4 +202,8 @@ void cmTest::DefineProperties(cmake *cm)
"The directory from which the test executable will be called.",
"If this is not set it is called from the directory the test executable "
"is located in.");
+ cm->DefineProperty
+ ("GENERATE_HTML", cmProperty::TEST,
+ "If set to true, this will generate HTML tags for passed and failed ",
+ "tests. If not set or not used, standard output will be used.");
}
--
1.7.10.4
Index: testDetails.php
===================================================================
--- testDetails.php (revision 3322)
+++ testDetails.php (working copy)
@@ -226,6 +226,14 @@
$xml .= add_XML_value("output", utf8_for_xml($testRow['output']));
}
+$outputIsHtml = 0; // default is plaintext
+$pos = strpos(@$uncompressedrow, "</span>");
+if($pos) // if span tag is found
+ {
+ $outputIsHtml = 1;
+ }
+$xml .= add_XML_value("outputtype", $outputIsHtml);
+
$xml .= add_XML_value("summaryLink", $summaryLink);
switch($testRow["status"])
{
Index: testDetails.xsl
===================================================================
--- testDetails.xsl (revision 3322)
+++ testDetails.xsl (working copy)
@@ -188,7 +188,15 @@
<br/>
<b>Test output</b>
<pre>
+<!-- To enable HTML output disable-output-escaping -->
+<xsl:choose>
+<xsl:when test="/cdash/test/outputtype=1">
+ <xsl:value-of disable-output-escaping="yes" select="cdash/test/output"/>
+</xsl:when>
+<xsl:otherwise>
<xsl:value-of select="cdash/test/output"/>
+</xsl:otherwise>
+</xsl:choose>
</pre>
<!-- FOOTER -->
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
