[cmake-developers] Custom commands with Ninja on Windows

2016-05-18 Thread Martin Ankerl 
Hi!

I've run into the same problem with many custom commands as discussed here:
http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/13434
Which is referenced in this bug report:
https://cmake.org/Bug/view.php?id=15612#c38924

I've had a look at this problem because this is the only thing that blocks
us from using ninja.

It seems to me that there are two possible solution:


* use a response file (rspfile and rspfile_content). This unfortunately is
only possible when modifying ninja to support multiline response files. As
far as I know this is not possible now. (the $in_newline does not work in
this case)


* Add a build target for each custom command, e.g. instead of

E.g. instead of an output like this:

build CMakeFiles\copy_czmq_3.0.2.util: CUSTOM_COMMAND
  COMMAND = cmd.exe /C "cd /D C:\dev\ras_robvis\vc140_x64_ninja &&
"c:\Program Files (x86)\CMake\bin\cmake.exe" -E copy_if_different
"C:/dev/ras_robvis/src/ext/lib_czmq/3.0.2/vc140_x64/czmq.dll"
"C:/dev/ras_robvis/vc140_x64_ninja/bin/./czmq.dll" && cd /D
C:\dev\ras_robvis\vc140_x64_ninja && "c:\Program Files
(x86)\CMake\bin\cmake.exe" -E copy_if_different
"C:/dev/ras_robvis/src/ext/lib_czmq/3.0.2/vc140_x64/czmqd.dll"
"C:/dev/ras_robvis/vc140_x64_ninja/bin/./czmqd.dll""
  DESC = Running utility command for copy_czmq_3.0.2
  restat = 1
build copy_czmq_3.0.2: phony CMakeFiles\copy_czmq_3.0.2.util


produce an output like this:

build CMakeFiles\copy_czmq_3.0.2.util.1: CUSTOM_COMMAND
  COMMAND = cmd.exe /C "cd /D C:\dev\ras_robvis\vc140_x64_ninja &&
"c:\Program Files (x86)\CMake\bin\cmake.exe" -E copy_if_different
"C:/dev/ras_robvis/src/ext/lib_czmq/3.0.2/vc140_x64/czmq.dll"
"C:/dev/ras_robvis/vc140_x64_ninja/bin/./czmq.dll""
  restat = 1

build CMakeFiles\copy_czmq_3.0.2.util.2: CUSTOM_COMMAND
  COMMAND = cmd.exe /C "cd /D C:\dev\ras_robvis\vc140_x64_ninja &&
"c:\Program Files (x86)\CMake\bin\cmake.exe" -E copy_if_different
"C:/dev/ras_robvis/src/ext/lib_czmq/3.0.2/vc140_x64/czmqd.dll"
"C:/dev/ras_robvis/vc140_x64_ninja/bin/./czmqd.dll""
  restat = 1

build CMakeFiles\copy_czmq_3.0.2.util: phony |
CMakeFiles\copy_czmq_3.0.2.util.1 CMakeFiles\copy_czmq_3.0.2.util.2
  DESC = Running utility command for copy_czmq_3.0.2

build copy_czmq_3.0.2: phony CMakeFiles\copy_czmq_3.0.2.util



I did not yet have a look at the cmake code so I am not sure how difficult
it is to implement this, or even if this is the best way to solve this.

Any comments? is this reasonable?

Martin

-- 
Martin
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [CMake 0016105]: Address "RegularExpression::compile(): Expression too big."

2016-05-18 Thread Mantis Bug Tracker 

The following issue has been SUBMITTED. 
== 
https://cmake.org/Bug/view.php?id=16105 
== 
Reported By:Roscoe A. Bartlett
Assigned To:
== 
Project:CMake
Issue ID:   16105
Category:   CMake
Reproducibility:always
Severity:   major
Priority:   normal
Status: new
== 
Date Submitted: 2016-05-18 20:13 EDT
Last Modified:  2016-05-18 20:13 EDT
== 
Summary:Address "RegularExpression::compile(): Expression
too big."
Description: 
For large, complex CMake projects, we are getting the error
"RegularExpression::compile(): Expression too big.".  This occurs when we can't
control the build directory path to make it short (i.e. $HOME/BUILD).  The below
email chain describes a situation where they are basically stuck.



From: Bartlett, Roscoe A 
Sent: Wednesday, May 18, 2016 7:52 PM
To: 'Johnson, Seth R.'
Cc: Evans, Thomas M.; Lefebvre, Robert A.
Subject: RE: [EXTERNAL] "RegularExpression::compile(): Expression too big."

Seth,

Yes, this is a CMake thing.  For CASL VERA, as you have said, we have gotten
away with a very shallow directory.  See:

https://tribits.org/doc/TribitsBuildReference.html#installing 

But you can also avoid this by disabling RPATH or going with static libraries
(both are bad options obviously).

I have it in the Kitware backlog to resolve this somehow:

   
https://docs.google.com/document/d/1TLHRp8eTNKw7udOhwIxrOYShXQUbxAzsXeOq5cwWnKM/edit#bookmark=id.a9drhmfrc9i1

but there are several higher priority things to do first (at least for CASL
VERA).

There was a PR for TriBITS a while back that was supposed to help this in some
situations:

   https://github.com/TriBITSPub/TriBITS/pull/79

but I don’t know if there is any more that can be done in TriBITS (do you see
any duplicate libraries?).

Looking at the CMake source code, it looks like they it hard-coded a 32 bit
integer range in a long (32767L) at:

  
https://github.com/Kitware/CMake/blob/master/Source/kwsys/RegularExpression.cxx#L372

You might see if that can be expanded to a 64 bit integer (which would remove
any limitation).

Cheers,

-Ross

P.S. How is Scale/Exnihilo connected to Spack?  I someone creating a package
install script for Scale/Exnihilo for Spack?


From: Johnson, Seth R. [mailto:johnso...@ornl.gov] 
Sent: Wednesday, May 18, 2016 4:00 PM
To: Bartlett, Roscoe A
Cc: Evans, Thomas M.; Lefebvre, Robert A.
Subject: [EXTERNAL] "RegularExpression::compile(): Expression too big."

Ross, 

I'm hitting the dreaded 

RegularExpression::compile(): Expression too big.
CMake Error at packages/ScalePrograms/cmake_install.cmake:45 (file):
  file RPATH_CHANGE could not write new RPATH:

error when compiling the full Exnihilo+SCALE on the new CADES machine.  I have
little control over the compile directory; I have no control over the wacky
Spack directories; and the number of libraries required to build SCALE is huge
because of the number of subpackages in Trilinos/Exnihilo/SCALE (49 Trilinos, 46
Exnihilo, 120 SCALE, 7 system/TPL).

Is there any solution aside from "try building with a shorter directory name and
hope that it's enough and hope that the number of packages doesn't grow"?

Thanks,
Seth

P.S.
The failing regex (with colons replaced by newlines) is:
/software/tools/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-5.3.0-5hy3c4b3xqemygnfwyl5dsc753gbvzrc/lib
/software/tools/spack/opt/spack/linux-x86_64/gcc-4.8.5/gcc-5.3.0-5hy3c4b3xqemygnfwyl5dsc753gbvzrc/lib64
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/ScaleDriver
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/Origen/origami
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/Origen/arp
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/Origen/couple
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/Origen/origen
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/Origen/SONInputLib
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/CsasShift
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/TKeno
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/PolarisWrapper
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/External
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/TNewt
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/Tsunami3D
/localscratch/compile/s3j-gcc5/Exnihilo-and-scale/packages/Sequence/Tsunami1D

Re: [cmake-developers] [Patch] ExternalProject: fix retry logic if error occurs

2016-05-18 Thread Brad King 
On 05/18/2016 03:30 PM, Ruslan Baratov via cmake-developers wrote:
> I've attached patch with applying retry logic in cases when status code 
> is not zero.

Thanks.  Please split the patch to perform the refactoring into
the ExternalProject-$step.cmake.in files first and then make the
logic changes as a second patch.  Also please add an explanation
of the logic change to the second commit message similar to what
you wrote in the original email.

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] [Patch] ExternalProject: fix retry logic if error occurs

2016-05-18 Thread Ruslan Baratov via cmake-developers 

Hi,

For now download retry attempt occurs in ExternalProject only if hash of 
archive mismatch. If error happens then script stop with FATAL_ERROR 
(example: 
https://ci.appveyor.com/project/ingenue/hunter/build/1.0.448/job/22kdl633f9w503ut). 
I've attached patch with applying retry logic in cases when status code 
is not zero.


Quite the same happening in 
https://github.com/Kitware/CMake/commit/30a94eecdb5c580d83a224848b78d186643e8105 
fix. Since `if(NOT status_code EQUAL 0)` trigger FATAL_ERROR, retry 
logic in `_ep_write_verifyfile_script` function is not reached. I've set 
default retry number to 5 with pauses 0, 5, 5, 15, 60 seconds. I left 
some space for future improvements if needed (90, 300, 1200=20min) 
probably can be controlled by some variable that will be available for 
users.


Ruslo
>From ce18b27a792ebbb79822b610e6d8e5077007d6cb Mon Sep 17 00:00:00 2001
From: Ruslan Baratov 
Date: Wed, 18 May 2016 22:06:35 +0300
Subject: [PATCH] ExternalProject: fix retry logic

Retry download if error occurs. Refactor ExternalProject module.
---
 Modules/ExternalProject-download.cmake.in | 161 ++
 Modules/ExternalProject-verify.cmake.in   |  48 +
 Modules/ExternalProject.cmake | 155 ++--
 3 files changed, 264 insertions(+), 100 deletions(-)
 create mode 100644 Modules/ExternalProject-download.cmake.in
 create mode 100644 Modules/ExternalProject-verify.cmake.in

diff --git a/Modules/ExternalProject-download.cmake.in b/Modules/ExternalProject-download.cmake.in
new file mode 100644
index 000..5b73cd8
--- /dev/null
+++ b/Modules/ExternalProject-download.cmake.in
@@ -0,0 +1,161 @@
+#=
+# Copyright 2008-2013 Kitware, Inc.
+# Copyright 2016 Ruslan Baratov
+#
+# Distributed under the OSI-approved BSD License (the "License");
+# see accompanying file Copyright.txt for details.
+#
+# This software is distributed WITHOUT ANY WARRANTY; without even the
+# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+# See the License for more information.
+#=
+# (To distribute this file outside of CMake, substitute the full
+#  License text for the above reference.)
+
+cmake_minimum_required(VERSION 3.5)
+
+function(check_file_hash has_hash hash_is_good)
+  if("${has_hash}" STREQUAL "")
+message(FATAL_ERROR "has_hash Can't be empty")
+  endif()
+
+  if("${hash_is_good}" STREQUAL "")
+message(FATAL_ERROR "hash_is_good Can't be empty")
+  endif()
+
+  if("@ALGO@" STREQUAL "")
+# No check
+set("${has_hash}" FALSE PARENT_SCOPE)
+set("${hash_is_good}" FALSE PARENT_SCOPE)
+return()
+  endif()
+
+  set("${has_hash}" TRUE PARENT_SCOPE)
+
+  message(STATUS "verifying file...
+   file='@LOCAL@'")
+
+  file("@ALGO@" "@LOCAL@" actual_value)
+
+  if(NOT "${actual_value}" STREQUAL "@EXPECT_VALUE@")
+set("${hash_is_good}" FALSE PARENT_SCOPE)
+message(STATUS "@ALGO@ hash of
+@LOCAL@
+  does not match expected value
+expected: '@EXPECT_VALUE@'
+  actual: '${actual_value}'")
+  else()
+set("${hash_is_good}" TRUE PARENT_SCOPE)
+  endif()
+endfunction()
+
+function(sleep_before_download attempt)
+  if(attempt EQUAL 0)
+return()
+  endif()
+
+  if(attempt EQUAL 1)
+message(STATUS "Retrying...")
+return()
+  endif()
+
+  set(sleep_seconds 0)
+
+  if(attempt EQUAL 2)
+set(sleep_seconds 5)
+  elseif(attempt EQUAL 3)
+set(sleep_seconds 5)
+  elseif(attempt EQUAL 4)
+set(sleep_seconds 15)
+  elseif(attempt EQUAL 5)
+set(sleep_seconds 60)
+  elseif(attempt EQUAL 6)
+set(sleep_seconds 90)
+  elseif(attempt EQUAL 7)
+set(sleep_seconds 300)
+  else()
+set(sleep_seconds 1200)
+  endif()
+
+  message(STATUS "Retry after ${sleep_seconds} seconds (attempt #${attempt}) ...")
+
+  execute_process(COMMAND "${CMAKE_COMMAND}" -E sleep "${sleep_seconds}")
+endfunction()
+
+if("@LOCAL@" STREQUAL "")
+  message(FATAL_ERROR "LOCAL can't be empty")
+endif()
+
+if("@REMOTE@" STREQUAL "")
+  message(FATAL_ERROR "REMOTE can't be empty")
+endif()
+
+if(EXISTS "@LOCAL@")
+  check_file_hash(has_hash hash_is_good)
+  if(has_hash)
+if(hash_is_good)
+  message(STATUS "File already exists and hash match (skip download):
+  file='@LOCAL@'
+  @ALGO@='@EXPECT_VALUE@'"
+  )
+  return()
+else()
+  message(STATUS "File already exists but hash mismatch. Removing...")
+  file(REMOVE "@LOCAL@")
+endif()
+  else()
+message(STATUS "File already exists but no hash specified (use URL_HASH):
+  file='@LOCAL@'
+Old file will be removed and new file downloaded from URL."
+)
+file(REMOVE "@LOCAL@")
+  endif()
+endif()
+
+set(retry_number 5)
+
+foreach(i RANGE ${retry_number})
+  sleep_before_download(${i})
+
+  message(STATUS "downloading...
+   src='@REMOTE@'
+   dst='@LOCAL@'
+

Re: [cmake-developers] [PATCH] CPackIFW: Added cpack_ifw_update_repository command and other

2016-05-18 Thread Konstantin Podsvirov 
18.05.2016, 16:52, "Brad King" :
> On 05/17/2016 04:29 PM, Konstantin Podsvirov wrote:
>> Please consider my changes in part CPack IFW generator.
>
> I've applied the changes with a few tweaks and merged to 'next' for testing:
>
> CPackIFW: Add more known QtIFW releases
> https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=6ca6b0dd
>
> CPackIFW: Add support for Promoting Updates
> https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=41199f8c

Thank you!

>> http://git.podsvirov.pro/?p=kitware/cmake.git;a=shortlog;h=refs/heads/topic-cpackifw-generator
>
> I'm unable to fetch from that repository anymore:
>
> $ git ls-remote git://podsvirov.pro/kitware/cmake.git
> fatal: unable to connect to podsvirov.pro:
> podsvirov.pro[0: 188.120.254.166]: errno=Connection refused
>
> What is the fetch URL for it?

All right now. This is the correct URL. Just my git-daemon forgot to start :-) 
Thanks for the notice.

> Thanks,
> -Brad

Regards,
Konstantin Podsvirov
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] [PATCH] CPackIFW: Added cpack_ifw_update_repository command and other

2016-05-18 Thread Brad King 
On 05/17/2016 04:29 PM, Konstantin Podsvirov wrote:
> Please consider my changes in part CPack IFW generator.

I've applied the changes with a few tweaks and merged to 'next' for testing:

 CPackIFW: Add more known QtIFW releases
 https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=6ca6b0dd

 CPackIFW: Add support for Promoting Updates
 https://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=41199f8c

> http://git.podsvirov.pro/?p=kitware/cmake.git;a=shortlog;h=refs/heads/topic-cpackifw-generator

I'm unable to fetch from that repository anymore:

 $ git ls-remote git://podsvirov.pro/kitware/cmake.git
 fatal: unable to connect to podsvirov.pro:
 podsvirov.pro[0: 188.120.254.166]: errno=Connection refused

What is the fetch URL for it?

Thanks,
-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers