Re: [cmake-developers] Fw: CTest and Address Sanitizer

[Karl Robinson]

Hi Chuck


Firstly, thanks for taking the time to respond.


The sanitizers exit on failure thing makes perfect sense although why it isn't 
the default behavior mystifies me. Turning it on gets me the effect I am 
looking for (CTest can detect a failure) but what I really want to know and the 
reason for my original post is:


1) Why do I always get Valgrind when I run memtest? Either that or "Memory 
checker (MemoryCheckCommand) not set, or cannot find the specified program.". 
Looking at the source code I _think_ setting  the MEMORYCHECK_TYPE variable to 
AddressSanitizer should be enough, I shouldn't need to set the 
MemoryCheckCommand and there isn't one for the sanitizers anyway


2) The documentation hints that CTest should be able to understand the output 
of the sanitizers, I can't say if this true or not with any certainty but it 
doesn't seem to be the case


Thanks again!



From: Chuck Atkins <chuck.atk...@kitware.com>
Sent: Thursday, January 5, 2017 11:47:13 AM
To: Karl Robinson
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] Fw: CTest and Address Sanitizer

Hi Karl,

>> I believe that the AddressSanitizer should normally stop with a non-zero 
>> error code right after the first error.

I have many tests within each executable test application, forcing the 
application to exit on the first error is not optimal since it means I cannot 
get an overview of all the issues and triage/fix them in logical order. Instead 
I have to test/fix/retest every time


That's by design and isn't to do with how CMake integrates asan; i.e. it's just 
how asan works.  The idea being that once you hit a memory corruption then any 
further results can't be relied on.  From the asan documentation: 
http://clang.llvm.org/docs/AddressSanitizer.html

"If a bug is detected, the program will print an error message to stderr and 
exit with a non-zero exit code. AddressSanitizer exits on the first detected 
error. This is by design:

  *   This approach allows AddressSanitizer to produce faster and smaller 
generated code (both by ~5%).
  *   Fixing bugs becomes unavoidable. AddressSanitizer does not produce false 
alarms. Once a memory corruption occurs, the program is in an inconsistent 
state, which could lead to confusing results and potentially misleading 
subsequent reports.

"

- Chuck


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Fw: CTest and Address Sanitizer

[Karl Robinson]

>> Maybe this blog post can be useful to you: 
>> https://blog.kitware.com/ctest-cdash-add-support-for-new-dynamic-analysis-tools/


Thanks Xavier, I did stumble across that blog post previously. It doesn't 
really help with my problem though. 


>> I believe that the AddressSanitizer should normally stop with a non-zero 
>> error code right after the first error.

I have many tests within each executable test application, forcing the 
application to exit on the first error is not optimal since it means I cannot 
get an overview of all the issues and triage/fix them in logical order. Instead 
I have to test/fix/retest every time

I should make some experiments with Valgrind, iirc ctest does understand 
Valgrind output. Of course, I already fixed all the issues Valgrind detected so 
now I have to fabri-cobble some new ones ⃣

I guess no one knows why Valgrind always seems to run when memtest is selected?

Thanks again for taking the time!

  


On Tue, Dec 27, 2016 at 1:05 AM, Karl Robinson   wrote:
 Hi

I am trying to use Address Sanitizer (also UndefinedBehaviour and Memory 
Sanitizers) with CTest (CMake 3.5.1).  In my top level cmake file I have

set(MEMORYCHECK_TYPE "AddressSanitizer") 
include (CTest)
enable_testing()
add_test( some_test_case )

Then I run my tests with 

        ctest -T memcheck


Whatever I do I always end up with Valgrind being run, occasionally (depending 
on what I have been messing with) CTest reports "Memory checker 
(MemoryCheckCommand) not set, or cannot find the specified program.")

In both cases my Dart config file looks like this

# Dynamic analysis (MemCheck)
PurifyCommand:
ValgrindCommand:
ValgrindCommandOptions:
MemoryCheckType: AddressSanitizer
MemoryCheckSanitizerOptions:
MemoryCheckCommand: /usr/bin/valgrind
MemoryCheckCommandOptions:
MemoryCheckSuppressionFile:

When I look through the cmake source code it seems that setting the 
MEMORYCHECK_TYPE variable to AddressSanitizer but with no MEMORYCHECK_COMMAND 
specified is correct. Why do  I always end up with Valgrind being selected? Is 
this the correct behavior?

When I run 'ctest -T test' on code built with a sanitizer enabled I can see 
bugs in the code firing the sanitizer and useful output is produced. So I don't 
understand why  Valgrind is required

The reason for trying to do this is that I was hoping that when I run a 
memcheck ctest would see the output from the sanitizers and mark the test case 
as failed. However, neither 'test' nor 'memcheck' detect the sanitizer failure

Any ideas what I might be doing wrong? Better, is there an example of using the 
sanitizers with ctest?

CMake & CTest are great tools but would be even more awesome if I could 
integrate CTest and the sanitizers

Thanks!


--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at:  
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support:  http://cmake.org/cmake/help/support.html
CMake Consulting:  http://cmake.org/cmake/help/consulting.html
CMake Training Courses:  http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at  
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
 



 -- 


Dr Xavier BESSERON
Research associate
FSTC, University of Luxembourg
Campus Kirchberg, Office E-007
Phone: +352 46 66 44 5418

http://luxdem.uni.lu/


   -- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Fw: CTest and Address Sanitizer

[Xavier Besseron]

Hi Karl,

Maybe this blog post can be useful to you:
https://blog.kitware.com/ctest-cdash-add-support-for-new-dynamic-analysis-tools/

I believe that the AddressSanitizer should normally stop with a non-zero
error code right after the first error.
You can check the runtime flags to control this behavior:
https://github.com/google/sanitizers/wiki/AddressSanitizerFlags
https://github.com/google/sanitizers/wiki/SanitizerCommonFlags


Best regards,

Xavier


On Tue, Dec 27, 2016 at 1:05 AM, Karl Robinson 
wrote:

> Hi
>
> I am trying to use Address Sanitizer (also UndefinedBehaviour and Memory
> Sanitizers) with CTest (CMake 3.5.1).  In my top level cmake file I have
>
> set(MEMORYCHECK_TYPE "AddressSanitizer")
> include (CTest)
> enable_testing()
> add_test( some_test_case )
>
> Then I run my tests with
>
> ctest -T memcheck
>
>
> Whatever I do I always end up with Valgrind being run, occasionally
> (depending on what I have been messing with) CTest reports "Memory checker
> (MemoryCheckCommand) not set, or cannot find the specified program.")
>
> In both cases my Dart config file looks like this
>
> # Dynamic analysis (MemCheck)
> PurifyCommand:
> ValgrindCommand:
> ValgrindCommandOptions:
> MemoryCheckType: AddressSanitizer
> MemoryCheckSanitizerOptions:
> MemoryCheckCommand: /usr/bin/valgrind
> MemoryCheckCommandOptions:
> MemoryCheckSuppressionFile:
>
> When I look through the cmake source code it seems that setting the
> MEMORYCHECK_TYPE variable to AddressSanitizer but with
> no MEMORYCHECK_COMMAND specified is correct. Why do  I always end up with
> Valgrind being selected? Is this the correct behavior?
>
> When I run 'ctest -T test' on code built with a sanitizer enabled I can
> see bugs in the code firing the sanitizer and useful output is produced. So
> I don't understand why  Valgrind is required
>
> The reason for trying to do this is that I was hoping that when I run a
> memcheck ctest would see the output from the sanitizers and mark the test
> case as failed. However, neither 'test' nor 'memcheck' detect the sanitizer
> failure
>
> Any ideas what I might be doing wrong? Better, is there an example of
> using the sanitizers with ctest?
>
> CMake & CTest are great tools but would be even more awesome if I could
> integrate CTest and the sanitizers
>
> Thanks!
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers
>



-- 
Dr Xavier BESSERON
Research associate
FSTC, University of Luxembourg
Campus Kirchberg, Office E-007
Phone: +352 46 66 44 5418
http://luxdem.uni.lu/
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers