Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
As there is already a patch for such a genex in bug 15509 and the discussion in 5939 (both linked below in Brad's reply) started 7 years ago... It would be a great pity to let this issue seep away again. What can I do to help fixing this finally? Stefan -Original Message- From: cmake-developers [mailto:cmake-developers-boun...@cmake.org] On Behalf Of CHEVRIER, Marc Sent: Freitag, 28. August 2015 10:07 To: Brad King; cmake-developers@cmake.org Subject: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens Offering generator-expressions is a very good idea because it offers a global solution to this problem. Now regarding the name of the generator, may be TO_NATIVE_PATH is not judicious, (even if problems explained in http://www.cmake.org/Bug/view.php?id=5939 are, in my point of view, linked to platforms not well handled rather than wrong semantic of TO_NATIVE_PATH) but PATH_FOR_SHELL is, as well, ambiguous at least for two reasons: * commands defined in add_custom_command, for example, are directly executed without any shell evolved * A system can have multiple shells supporting different paths syntax. For example, on windows, my shell is tcsh which handle paths with ‘/‘ rather than ‘\' May be, TO_SYSTEM_PATH or TO_HOST_SYSTEM_PATH make more sense… Marc On 27/08/15 17:18, "cmake-developers on behalf of Brad King" <cmake-developers-boun...@cmake.org on behalf of brad.k...@kitware.com> wrote: >On 08/27/2015 10:06 AM, James Johnston wrote: >> I would vote naming it TO_NATIVE_PATH instead of PATH_FOR_SHELL, for >> consistency with the existing parameter in the file command. > >The file(TO_NATIVE_PATH) command is hopelessly ill-defined: > > http://www.cmake.org/Bug/view.php?id=5939 > >See also existing discussion about a genex for such conversions: > > http://www.cmake.org/Bug/view.php?id=15509 > >-Brad > >-- > >Powered by www.kitware.com > >Please keep messages on-topic and check the CMake FAQ at: >http://www.cmake.org/Wiki/CMake_FAQ > >Kitware offers various services to support the CMake community. For more >information on each offering, please visit: > >CMake Support: http://cmake.org/cmake/help/support.html >CMake Consulting: http://cmake.org/cmake/help/consulting.html >CMake Training Courses: http://cmake.org/cmake/help/training.html > >Visit other Kitware open-source projects at >http://www.kitware.com/opensource/opensource.html > >Follow this link to subscribe/unsubscribe: >http://public.kitware.com/mailman/listinfo/cmake-developers -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
Offering generator-expressions is a very good idea because it offers a global solution to this problem. Now regarding the name of the generator, may be TO_NATIVE_PATH is not judicious, (even if problems explained in http://www.cmake.org/Bug/view.php?id=5939 are, in my point of view, linked to platforms not well handled rather than wrong semantic of TO_NATIVE_PATH) but PATH_FOR_SHELL is, as well, ambiguous at least for two reasons: * commands defined in add_custom_command, for example, are directly executed without any shell evolved * A system can have multiple shells supporting different paths syntax. For example, on windows, my shell is tcsh which handle paths with ‘/‘ rather than ‘\' May be, TO_SYSTEM_PATH or TO_HOST_SYSTEM_PATH make more sense… Marc On 27/08/15 17:18, cmake-developers on behalf of Brad King cmake-developers-boun...@cmake.org on behalf of brad.k...@kitware.com wrote: On 08/27/2015 10:06 AM, James Johnston wrote: I would vote naming it TO_NATIVE_PATH instead of PATH_FOR_SHELL, for consistency with the existing parameter in the file command. The file(TO_NATIVE_PATH) command is hopelessly ill-defined: http://www.cmake.org/Bug/view.php?id=5939 See also existing discussion about a genex for such conversions: http://www.cmake.org/Bug/view.php?id=15509 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
This is pretty nice from the perspective of an engineer. But I wonder if it
wouldn't be kind of over-engineered in the sense that it would be a rather
hidden feature for the purpose? -- I probably wouldn't think of searching for a
generator expression in the documentation when I have trouble with the
replacement of directory tokens. Then again, there could be a hint in the
ExternalProject documentation which would be fine I guess. One way or another,
I would be happy if we could determine what the patch should provide exactly so
that we come to an end. :-) Thank you for all the feedback so far.
-Original Message-
From: Brad King [mailto:brad.k...@kitware.com]
Sent: Donnerstag, 27. August 2015 15:10
To: Kislinskiy, Stefan; CHEVRIER, Marc; David Cole
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as substitute
for directory tokens
On 08/26/2015 03:35 AM, Kislinskiy, Stefan wrote:
Would you prefer to have a switch for each *_DIR variable for all
target steps, or a common switch but for each target step, like the
new USE_TERMINAL switches in the master?
[snip]
Von: CHEVRIER, Marc [marc.chevr...@sap.com] Offering the possibility
to manage native paths in an easy way is a very good enhancement
Another approach is to introduce a generator expression to transform the path
slash style. Then it could be used both for ExternalProject and in other
custom commands. E.g.
$PATH_FOR_SHELL:SOURCE_DIR/bootstrap${shell_ext}
In this case SOURCE_DIR would be replaced with forward slashes by
ExternalProject but then at generate time CMake would evaluate the genex to
convert slashes as needed.
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
On 08/26/2015 03:35 AM, Kislinskiy, Stefan wrote:
Would you prefer to have a switch for each *_DIR variable for all
target steps, or a common switch but for each target step, like
the new USE_TERMINAL switches in the master?
[snip]
Von: CHEVRIER, Marc [marc.chevr...@sap.com]
Offering the possibility to manage native paths in an easy way
is a very good enhancement
Another approach is to introduce a generator expression to transform
the path slash style. Then it could be used both for ExternalProject
and in other custom commands. E.g.
$PATH_FOR_SHELL:SOURCE_DIR/bootstrap${shell_ext}
In this case SOURCE_DIR would be replaced with forward slashes
by ExternalProject but then at generate time CMake would evaluate
the genex to convert slashes as needed.
-Brad
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
Yes, TO_CMAKE_PATH is used quite often. Just imagine parsing paths as input. Or
when you create CMake scripts during the build you might want to call
file(TO_CMAKE_PATH) in there. It makes sense on both ends, as input and output.
:) I also vote for consistent naming, if the generator expressions are chosen
as the final solution.
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org] On Behalf Of
James Johnston
Sent: Donnerstag, 27. August 2015 16:06
To: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as substitute
for directory tokens
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
On Behalf Of Kislinskiy, Stefan
Sent: Thursday, August 27, 2015 13:44
To: Brad King; CHEVRIER, Marc; David Cole
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
This is pretty nice from the perspective of an engineer. But I wonder
if
it
wouldn't be kind of over-engineered in the sense that it would be a
rather hidden feature for the purpose? -- I probably wouldn't think of
searching
for
a generator expression in the documentation when I have trouble with
the replacement of directory tokens. Then again, there could be a hint
in the ExternalProject documentation which would be fine I guess. One
way or another, I would be happy if we could determine what the patch
should provide exactly so that we come to an end. :-) Thank you for
all the
feedback
so far.
I think Brad raises an excellent point here though I hadn't thought of...
implementing the underlying functionality as a generator expression allows the
problem to be solved outside of the scope of ExternalProject when it arises...
For example if I add_custom_command as a custom build step and need to pass a
native path argument to it - a path known only at generate time. I guess it is
not possible right now?
Another approach is to introduce a generator expression to transform
the path slash style. Then it could be used both for ExternalProject
and in
other
custom commands. E.g.
$PATH_FOR_SHELL:SOURCE_DIR/bootstrap${shell_ext}
In this case SOURCE_DIR would be replaced with forward slashes by
ExternalProject but then at generate time CMake would evaluate the
genex to convert slashes as needed.
I would vote naming it TO_NATIVE_PATH instead of PATH_FOR_SHELL, for
consistency with the existing parameter in the file command. There could then
also be a corresponding TO_CMAKE_PATH gen-exp, although I wonder if it would be
used much. (Then again, do people even use file(TO_CMAKE_PATH) very much?)
James
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
On Behalf Of Kislinskiy, Stefan
Sent: Thursday, August 27, 2015 13:44
To: Brad King; CHEVRIER, Marc; David Cole
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
This is pretty nice from the perspective of an engineer. But I wonder if
it
wouldn't be kind of over-engineered in the sense that it would be a rather
hidden feature for the purpose? -- I probably wouldn't think of searching
for
a generator expression in the documentation when I have trouble with the
replacement of directory tokens. Then again, there could be a hint in the
ExternalProject documentation which would be fine I guess. One way or
another, I would be happy if we could determine what the patch should
provide exactly so that we come to an end. :-) Thank you for all the
feedback
so far.
I think Brad raises an excellent point here though I hadn't thought of...
implementing the underlying functionality as a generator expression allows
the problem to be solved outside of the scope of ExternalProject when it
arises... For example if I add_custom_command as a custom build step and
need to pass a native path argument to it - a path known only at generate
time. I guess it is not possible right now?
Another approach is to introduce a generator expression to transform the
path slash style. Then it could be used both for ExternalProject and in
other
custom commands. E.g.
$PATH_FOR_SHELL:SOURCE_DIR/bootstrap${shell_ext}
In this case SOURCE_DIR would be replaced with forward slashes by
ExternalProject but then at generate time CMake would evaluate the genex
to convert slashes as needed.
I would vote naming it TO_NATIVE_PATH instead of PATH_FOR_SHELL, for
consistency with the existing parameter in the file command. There could
then also be a corresponding TO_CMAKE_PATH gen-exp, although I wonder if it
would be used much. (Then again, do people even use file(TO_CMAKE_PATH)
very much?)
James
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
On 08/27/2015 10:06 AM, James Johnston wrote: I would vote naming it TO_NATIVE_PATH instead of PATH_FOR_SHELL, for consistency with the existing parameter in the file command. The file(TO_NATIVE_PATH) command is hopelessly ill-defined: http://www.cmake.org/Bug/view.php?id=5939 See also existing discussion about a genex for such conversions: http://www.cmake.org/Bug/view.php?id=15509 -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
I agree with David. Offering the possibility to manage native paths in an easy way is a very good enhancement (Today, I rely on some specific actions when I am on Win32 to manage native path for ONE specific step of ExternalProject) BUT, offering only the alternative to have NONE or ALL paths in native form is, by far, too restrictive. So I am for the solution providing new patterns as alternative to current ones to manage native paths, i.e. Adding *_NATIVE_DIR for all already available *_DIR patterns. Marc On 25/08/15 17:31, cmake-developers on behalf of David Cole via cmake-developers cmake-developers-boun...@cmake.org on behalf of cmake-developers@cmake.org wrote: I'm going to let other CMake developers chime in on this one. It's better than the first patch, because it's opt-in, and you have to add something to get different than previous behavior, but I'm still of the opinion that this is very command specific, and you won't always want all _DIR references as native. In my opinion, it's better left to the person constructing the ExternalProject_Add call. But I am curious to hear other CMake devs give their opinions. David C. On Tue, Aug 25, 2015 at 3:18 AM, Kislinskiy, Stefan s.kislins...@dkfz-heidelberg.de wrote: Dear CMake developers, any thoughts on the fix? :) Best regards, Stefan Kislinskiy Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von Kislinskiy, Stefan [s.kislins...@dkfz-heidelberg.de] Gesendet: Freitag, 21. August 2015 23:56 An: David Cole; James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens What do you think about the new patch I attached to this mail? It adds an option NATIVE_DIR_TOKENS to ExternalProjects_Add. I also attached a CMake script file which tests/shows this feature. Stefan Kislinskiy Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von David Cole via cmake-developers [cmake-developers@cmake.org] Gesendet: Donnerstag, 20. August 2015 23:20 An: James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens It's exactly what I am concerned about: You're asking to change the behavior of something for EVERYONE to solve a problem which you have encountered. If you change it the way you have proposed, you will cause problems for others. It has worked the way it is now since ExternalProject_Add was introduced in CMake 2.8. Changing it unconditionally the way you propose is simply not feasible for backwards compatibility. I think commands that take native paths ought NOT to use the *_DIR replacement values, and instead, ought to pass in variables that contain the native paths in the first place. David C. On Thu, Aug 20, 2015 at 2:58 PM, James Johnston johnstonj.pub...@codenest.com wrote: Hi, Funny you are mailing the list about this, since I just ran into this same issue today building something totally different from Boost... In this case it's a build tool that thinks the /U in C:/Users is a new command line argument, that isn't recognized - and then the subsequent s also ends up unrecognized... and it all fails... And it has nothing to do with the working directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible workaround for me. I think the issue with globally making this change to the existing tokens is that there could be some external tool/program that is EXPECTING to get CMake paths, not native paths. Who knows? I am guessing that is what David Cole was concerned about. Maybe the right answer is to introduce some NEW tokens while leaving the behavior of the old ones unchanged. E.g. BINARY_DIR_NATIVE etc. It would be good if the patch updates the documentation of ExternalProject and clearly states the path format of BINARY_DIR vs BINARY_DIR_NATIVE. Then the user can pick whichever one suits them best, depending on the tool being invoked. Furthermore, sometimes BINARY_DIR_NATIVE still needs to be replaced with a CMake path, not native path. For example, if the token is being found in a property like WORKING_DIRECTORY that eventually gets passed to add_custom_command(WORKING_DIRECTORY) then I'm guessing still has to be a CMake path. I am guessing this is what David Cole was also concerned about. I still think your original method of building Boost is a bit unusual and would be better served by _Add_Step with a custom working directory - because that's the publicly documented/standard way of changing the working directory, but that is up to you. :) Best regards, James Johnston On Thu, 20 Aug 2015 14:37:08 + Stefan Kislinskiy s.kislins...@dkfz-heidelberg.de wrote Hi, thank you for our suggestions. I am aware
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
Would you prefer to have a switch for each *_DIR variable for all target steps, or a common switch but for each target step, like the new USE_TERMINAL switches in the master? Von: CHEVRIER, Marc [marc.chevr...@sap.com] Gesendet: Mittwoch, 26. August 2015 08:49 An: David Cole; Kislinskiy, Stefan Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens I agree with David. Offering the possibility to manage native paths in an easy way is a very good enhancement (Today, I rely on some specific actions when I am on Win32 to manage native path for ONE specific step of ExternalProject) BUT, offering only the alternative to have NONE or ALL paths in native form is, by far, too restrictive. So I am for the solution providing new patterns as alternative to current ones to manage native paths, i.e. Adding *_NATIVE_DIR for all already available *_DIR patterns. Marc On 25/08/15 17:31, cmake-developers on behalf of David Cole via cmake-developers cmake-developers-boun...@cmake.org on behalf of cmake-developers@cmake.org wrote: I'm going to let other CMake developers chime in on this one. It's better than the first patch, because it's opt-in, and you have to add something to get different than previous behavior, but I'm still of the opinion that this is very command specific, and you won't always want all _DIR references as native. In my opinion, it's better left to the person constructing the ExternalProject_Add call. But I am curious to hear other CMake devs give their opinions. David C. On Tue, Aug 25, 2015 at 3:18 AM, Kislinskiy, Stefan s.kislins...@dkfz-heidelberg.de wrote: Dear CMake developers, any thoughts on the fix? :) Best regards, Stefan Kislinskiy Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von Kislinskiy, Stefan [s.kislins...@dkfz-heidelberg.de] Gesendet: Freitag, 21. August 2015 23:56 An: David Cole; James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens What do you think about the new patch I attached to this mail? It adds an option NATIVE_DIR_TOKENS to ExternalProjects_Add. I also attached a CMake script file which tests/shows this feature. Stefan Kislinskiy Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von David Cole via cmake-developers [cmake-developers@cmake.org] Gesendet: Donnerstag, 20. August 2015 23:20 An: James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens It's exactly what I am concerned about: You're asking to change the behavior of something for EVERYONE to solve a problem which you have encountered. If you change it the way you have proposed, you will cause problems for others. It has worked the way it is now since ExternalProject_Add was introduced in CMake 2.8. Changing it unconditionally the way you propose is simply not feasible for backwards compatibility. I think commands that take native paths ought NOT to use the *_DIR replacement values, and instead, ought to pass in variables that contain the native paths in the first place. David C. On Thu, Aug 20, 2015 at 2:58 PM, James Johnston johnstonj.pub...@codenest.com wrote: Hi, Funny you are mailing the list about this, since I just ran into this same issue today building something totally different from Boost... In this case it's a build tool that thinks the /U in C:/Users is a new command line argument, that isn't recognized - and then the subsequent s also ends up unrecognized... and it all fails... And it has nothing to do with the working directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible workaround for me. I think the issue with globally making this change to the existing tokens is that there could be some external tool/program that is EXPECTING to get CMake paths, not native paths. Who knows? I am guessing that is what David Cole was concerned about. Maybe the right answer is to introduce some NEW tokens while leaving the behavior of the old ones unchanged. E.g. BINARY_DIR_NATIVE etc. It would be good if the patch updates the documentation of ExternalProject and clearly states the path format of BINARY_DIR vs BINARY_DIR_NATIVE. Then the user can pick whichever one suits them best, depending on the tool being invoked. Furthermore, sometimes BINARY_DIR_NATIVE still needs to be replaced with a CMake path, not native path. For example, if the token is being found in a property like WORKING_DIRECTORY that eventually gets passed to add_custom_command(WORKING_DIRECTORY) then I'm guessing still has to be a CMake path. I am guessing this is what David Cole was also
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
I see. Added native-style path replacements for SOURCE_NATIVE_DIR and so on.
Also extended the documentation accordingly.
Von: CHEVRIER, Marc [marc.chevr...@sap.com]
Gesendet: Mittwoch, 26. August 2015 10:26
An: Kislinskiy, Stefan; David Cole
Cc: cmake-developers@cmake.org
Betreff: Re: AW: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
I didn’t even think about switches. I don’t think they are required.
Adding the capability to handle paths in native format seems enough.
Example: I want to use a specific tool for the build step which supports only
native paths
ExternalProjet_add (
….
SOURCE_DIR c:/sources/my_project
….
BUILD_COMMAND ${MY_CUSTOM_COMMAND} -IN SOURCE_NATIVE_DIR -OUT
BINARY_NATIVE_DIR
…
)
With this approach, the previous behaviour is ensured and it is easy, for
specify cases, to use native paths.
On 26/08/15 09:35, Kislinskiy, Stefan s.kislins...@dkfz-heidelberg.de wrote:
Would you prefer to have a switch for each *_DIR variable for all target
steps, or a common switch but for each target step, like the new USE_TERMINAL
switches in the master?
Von: CHEVRIER, Marc [marc.chevr...@sap.com]
Gesendet: Mittwoch, 26. August 2015 08:49
An: David Cole; Kislinskiy, Stefan
Cc: cmake-developers@cmake.org
Betreff: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
I agree with David.
Offering the possibility to manage native paths in an easy way is a very good
enhancement (Today, I rely on some specific actions when I am on Win32 to
manage native path for ONE specific step of ExternalProject) BUT, offering
only the alternative to have NONE or ALL paths in native form is, by far, too
restrictive.
So I am for the solution providing new patterns as alternative to current ones
to manage native paths, i.e. Adding *_NATIVE_DIR for all already available
*_DIR patterns.
Marc
On 25/08/15 17:31, cmake-developers on behalf of David Cole via
cmake-developers cmake-developers-boun...@cmake.org on behalf of
cmake-developers@cmake.org wrote:
I'm going to let other CMake developers chime in on this one.
It's better than the first patch, because it's opt-in, and you have to
add something to get different than previous behavior, but I'm still
of the opinion that this is very command specific, and you won't
always want all _DIR references as native. In my opinion, it's better
left to the person constructing the ExternalProject_Add call. But I am
curious to hear other CMake devs give their opinions.
David C.
On Tue, Aug 25, 2015 at 3:18 AM, Kislinskiy, Stefan
s.kislins...@dkfz-heidelberg.de wrote:
Dear CMake developers,
any thoughts on the fix? :)
Best regards,
Stefan Kislinskiy
Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von
Kislinskiy, Stefan [s.kislins...@dkfz-heidelberg.de]
Gesendet: Freitag, 21. August 2015 23:56
An: David Cole; James Johnston
Cc: cmake-developers@cmake.org
Betreff: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
What do you think about the new patch I attached to this mail? It adds an
option NATIVE_DIR_TOKENS to ExternalProjects_Add. I also attached a CMake
script file which tests/shows this feature.
Stefan Kislinskiy
Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von
David Cole via cmake-developers [cmake-developers@cmake.org]
Gesendet: Donnerstag, 20. August 2015 23:20
An: James Johnston
Cc: cmake-developers@cmake.org
Betreff: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
It's exactly what I am concerned about:
You're asking to change the behavior of something for EVERYONE to
solve a problem which you have encountered. If you change it the way
you have proposed, you will cause problems for others. It has worked
the way it is now since ExternalProject_Add was introduced in CMake
2.8. Changing it unconditionally the way you propose is simply not
feasible for backwards compatibility.
I think commands that take native paths ought NOT to use the *_DIR
replacement values, and instead, ought to pass in variables that
contain the native paths in the first place.
David C.
On Thu, Aug 20, 2015 at 2:58 PM, James Johnston
johnstonj.pub...@codenest.com wrote:
Hi,
Funny you are mailing the list about this, since I just ran into this same
issue today building something totally different from Boost... In this
case it's a build tool that thinks the /U in C:/Users is a new command
line argument, that isn't recognized - and then the subsequent s also
ends up unrecognized... and it all fails... And it has nothing to do with
the working directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible
workaround for me
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
I didn’t even think about switches. I don’t think they are required.
Adding the capability to handle paths in native format seems enough.
Example: I want to use a specific tool for the build step which supports only
native paths
ExternalProjet_add (
….
SOURCE_DIR c:/sources/my_project
….
BUILD_COMMAND ${MY_CUSTOM_COMMAND} -IN SOURCE_NATIVE_DIR -OUT
BINARY_NATIVE_DIR
…
)
With this approach, the previous behaviour is ensured and it is easy, for
specify cases, to use native paths.
On 26/08/15 09:35, Kislinskiy, Stefan s.kislins...@dkfz-heidelberg.de wrote:
Would you prefer to have a switch for each *_DIR variable for all target
steps, or a common switch but for each target step, like the new USE_TERMINAL
switches in the master?
Von: CHEVRIER, Marc [marc.chevr...@sap.com]
Gesendet: Mittwoch, 26. August 2015 08:49
An: David Cole; Kislinskiy, Stefan
Cc: cmake-developers@cmake.org
Betreff: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
I agree with David.
Offering the possibility to manage native paths in an easy way is a very good
enhancement (Today, I rely on some specific actions when I am on Win32 to
manage native path for ONE specific step of ExternalProject) BUT, offering
only the alternative to have NONE or ALL paths in native form is, by far, too
restrictive.
So I am for the solution providing new patterns as alternative to current ones
to manage native paths, i.e. Adding *_NATIVE_DIR for all already available
*_DIR patterns.
Marc
On 25/08/15 17:31, cmake-developers on behalf of David Cole via
cmake-developers cmake-developers-boun...@cmake.org on behalf of
cmake-developers@cmake.org wrote:
I'm going to let other CMake developers chime in on this one.
It's better than the first patch, because it's opt-in, and you have to
add something to get different than previous behavior, but I'm still
of the opinion that this is very command specific, and you won't
always want all _DIR references as native. In my opinion, it's better
left to the person constructing the ExternalProject_Add call. But I am
curious to hear other CMake devs give their opinions.
David C.
On Tue, Aug 25, 2015 at 3:18 AM, Kislinskiy, Stefan
s.kislins...@dkfz-heidelberg.de wrote:
Dear CMake developers,
any thoughts on the fix? :)
Best regards,
Stefan Kislinskiy
Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von
Kislinskiy, Stefan [s.kislins...@dkfz-heidelberg.de]
Gesendet: Freitag, 21. August 2015 23:56
An: David Cole; James Johnston
Cc: cmake-developers@cmake.org
Betreff: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
What do you think about the new patch I attached to this mail? It adds an
option NATIVE_DIR_TOKENS to ExternalProjects_Add. I also attached a CMake
script file which tests/shows this feature.
Stefan Kislinskiy
Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von
David Cole via cmake-developers [cmake-developers@cmake.org]
Gesendet: Donnerstag, 20. August 2015 23:20
An: James Johnston
Cc: cmake-developers@cmake.org
Betreff: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
It's exactly what I am concerned about:
You're asking to change the behavior of something for EVERYONE to
solve a problem which you have encountered. If you change it the way
you have proposed, you will cause problems for others. It has worked
the way it is now since ExternalProject_Add was introduced in CMake
2.8. Changing it unconditionally the way you propose is simply not
feasible for backwards compatibility.
I think commands that take native paths ought NOT to use the *_DIR
replacement values, and instead, ought to pass in variables that
contain the native paths in the first place.
David C.
On Thu, Aug 20, 2015 at 2:58 PM, James Johnston
johnstonj.pub...@codenest.com wrote:
Hi,
Funny you are mailing the list about this, since I just ran into this same
issue today building something totally different from Boost... In this
case it's a build tool that thinks the /U in C:/Users is a new command
line argument, that isn't recognized - and then the subsequent s also
ends up unrecognized... and it all fails... And it has nothing to do with
the working directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible
workaround for me.
I think the issue with globally making this change to the existing tokens
is that there could be some external tool/program that is EXPECTING to get
CMake paths, not native paths. Who knows? I am guessing that is what
David Cole was concerned about.
Maybe the right answer is to introduce some NEW tokens while leaving the
behavior of the old ones unchanged. E.g. BINARY_DIR_NATIVE etc. It
would be good
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
-Original Message- From: cmake-developers [mailto:cmake-developers-boun...@cmake.org] On Behalf Of Kislinskiy, Stefan Sent: Wednesday, August 26, 2015 10:03 To: CHEVRIER, Marc; David Cole Cc: cmake-developers@cmake.org Subject: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens I see. Added native-style path replacements for SOURCE_NATIVE_DIR and so on. Also extended the documentation accordingly. Some relatively major/minor/nitpicky thoughts in the interests of thoroughness: * There's no test code. It would be good if there was some code to explicitly test these options for nightly testing. For that matter, I think the original tokens weren't thoroughly tested either (e.g. a quick grep for TMP_DIR doesn't show usage of that token at all in the tests). The whole situation could therefore use some help, I think. There are several existing tests of ExternalProject that could probably be extended to test these tokens as needed. * Token replacement is done for WORKING_DIRECTORY and BYPRODUCTS parameters of add_custom_command. You'd best be sure those parameters of add_custom_command can handle a native path, or else not convert to native paths - even for *_NATIVE tokens - when the path is destined for one of those parameters. (I'd tend to assume the command *could* handle it properly since a lot of users might inadvertently pass them, but it's something that should be double-checked. If the command is passing those paths as-is to the generator, correct operation could even vary by generator.) The easy way out might be to have a parameter on _ep_replace_location_tags declaring whether to ACTUALLY convert native path tokens to native paths, and then callers to that macro can decide if they can / should handle native paths. * It would be useful if the documentation explicitly stated that the old BINARY_DIR friends variables are using CMake paths; currently the documentation does not say - which was a shortcoming of the original docs (as you found out accidentally, it is CMake paths). * Nothing explicitly to do with your patch, but the documentation doesn't state that any of the tokens are valid in ExternalProject_Add; it may be useful to audit the options of ExternalProject_Add and see where they can be used, and document accordingly. (Some are obvious, like CONFIGURE_COMMAND, but some are less obvious, like CMAKE_CACHE_ARGS / CMAKE_CACHE_DEFAULT_ARGS which appear to have their own special handling for these tokens, yet none of that is documented. Even a newcomer might not realize the obvious, if they don't realize that ExternalProject_Add is implemented via ExternalProject_AddStep.) * New features should also be documented in Help/release/dev so that it makes it into the release notes of the next CMake version (i.e. so people notice your new feature). I leave it to others to decide if adding the new *_NATIVE tokens is the right way, or if there's a better way - as I myself am a newcomer to CMake. I think I like it though. :) Best regards, James Johnston -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
Dear CMake developers, any thoughts on the fix? :) Best regards, Stefan Kislinskiy Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von Kislinskiy, Stefan [s.kislins...@dkfz-heidelberg.de] Gesendet: Freitag, 21. August 2015 23:56 An: David Cole; James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens What do you think about the new patch I attached to this mail? It adds an option NATIVE_DIR_TOKENS to ExternalProjects_Add. I also attached a CMake script file which tests/shows this feature. Stefan Kislinskiy Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von David Cole via cmake-developers [cmake-developers@cmake.org] Gesendet: Donnerstag, 20. August 2015 23:20 An: James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens It's exactly what I am concerned about: You're asking to change the behavior of something for EVERYONE to solve a problem which you have encountered. If you change it the way you have proposed, you will cause problems for others. It has worked the way it is now since ExternalProject_Add was introduced in CMake 2.8. Changing it unconditionally the way you propose is simply not feasible for backwards compatibility. I think commands that take native paths ought NOT to use the *_DIR replacement values, and instead, ought to pass in variables that contain the native paths in the first place. David C. On Thu, Aug 20, 2015 at 2:58 PM, James Johnston johnstonj.pub...@codenest.com wrote: Hi, Funny you are mailing the list about this, since I just ran into this same issue today building something totally different from Boost... In this case it's a build tool that thinks the /U in C:/Users is a new command line argument, that isn't recognized - and then the subsequent s also ends up unrecognized... and it all fails... And it has nothing to do with the working directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible workaround for me. I think the issue with globally making this change to the existing tokens is that there could be some external tool/program that is EXPECTING to get CMake paths, not native paths. Who knows? I am guessing that is what David Cole was concerned about. Maybe the right answer is to introduce some NEW tokens while leaving the behavior of the old ones unchanged. E.g. BINARY_DIR_NATIVE etc. It would be good if the patch updates the documentation of ExternalProject and clearly states the path format of BINARY_DIR vs BINARY_DIR_NATIVE. Then the user can pick whichever one suits them best, depending on the tool being invoked. Furthermore, sometimes BINARY_DIR_NATIVE still needs to be replaced with a CMake path, not native path. For example, if the token is being found in a property like WORKING_DIRECTORY that eventually gets passed to add_custom_command(WORKING_DIRECTORY) then I'm guessing still has to be a CMake path. I am guessing this is what David Cole was also concerned about. I still think your original method of building Boost is a bit unusual and would be better served by _Add_Step with a custom working directory - because that's the publicly documented/standard way of changing the working directory, but that is up to you. :) Best regards, James Johnston On Thu, 20 Aug 2015 14:37:08 + Stefan Kislinskiy s.kislins...@dkfz-heidelberg.de wrote Hi, thank you for our suggestions. I am aware that I can solve my example differently and that it might look not directly connected the proposal, but well, it is an example just to show a single case and why it matters. :) I did not want to discuss the example itself. Working around here would just resolve a symptom. My point is the overall problem that would persist: A big part of ExternalProject is to issue commands for predefined and custom steps. Those commands are supposed to be executed by the shell/command line. According to the documentation and the source code of ExternalProject, directory tokens are mainly supposed to be replaced in commands. It is my understanding, that it is a bug, if CMake isn't able to assemble these commands correctly. This would include usage of the correct path style of the OS for shell/command line commands. As directory tokens are replaced internally right before a shell/command line command is assembled, I can't see why this would be kind of API-breaking. You cannot interfere in your CMake code with these internal replacements. Therefore I would still prefer my solution as it is pretty simple without adding even more features to ExternalProject and in my opinion without breaking code in the wild. It is a true bug fix instead of a feature
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
What do you think about the new patch I attached to this mail? It adds an option NATIVE_DIR_TOKENS to ExternalProjects_Add. I also attached a CMake script file which tests/shows this feature. Stefan Kislinskiy Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von David Cole via cmake-developers [cmake-developers@cmake.org] Gesendet: Donnerstag, 20. August 2015 23:20 An: James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens It's exactly what I am concerned about: You're asking to change the behavior of something for EVERYONE to solve a problem which you have encountered. If you change it the way you have proposed, you will cause problems for others. It has worked the way it is now since ExternalProject_Add was introduced in CMake 2.8. Changing it unconditionally the way you propose is simply not feasible for backwards compatibility. I think commands that take native paths ought NOT to use the *_DIR replacement values, and instead, ought to pass in variables that contain the native paths in the first place. David C. On Thu, Aug 20, 2015 at 2:58 PM, James Johnston johnstonj.pub...@codenest.com wrote: Hi, Funny you are mailing the list about this, since I just ran into this same issue today building something totally different from Boost... In this case it's a build tool that thinks the /U in C:/Users is a new command line argument, that isn't recognized - and then the subsequent s also ends up unrecognized... and it all fails... And it has nothing to do with the working directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible workaround for me. I think the issue with globally making this change to the existing tokens is that there could be some external tool/program that is EXPECTING to get CMake paths, not native paths. Who knows? I am guessing that is what David Cole was concerned about. Maybe the right answer is to introduce some NEW tokens while leaving the behavior of the old ones unchanged. E.g. BINARY_DIR_NATIVE etc. It would be good if the patch updates the documentation of ExternalProject and clearly states the path format of BINARY_DIR vs BINARY_DIR_NATIVE. Then the user can pick whichever one suits them best, depending on the tool being invoked. Furthermore, sometimes BINARY_DIR_NATIVE still needs to be replaced with a CMake path, not native path. For example, if the token is being found in a property like WORKING_DIRECTORY that eventually gets passed to add_custom_command(WORKING_DIRECTORY) then I'm guessing still has to be a CMake path. I am guessing this is what David Cole was also concerned about. I still think your original method of building Boost is a bit unusual and would be better served by _Add_Step with a custom working directory - because that's the publicly documented/standard way of changing the working directory, but that is up to you. :) Best regards, James Johnston On Thu, 20 Aug 2015 14:37:08 + Stefan Kislinskiy s.kislins...@dkfz-heidelberg.de wrote Hi, thank you for our suggestions. I am aware that I can solve my example differently and that it might look not directly connected the proposal, but well, it is an example just to show a single case and why it matters. :) I did not want to discuss the example itself. Working around here would just resolve a symptom. My point is the overall problem that would persist: A big part of ExternalProject is to issue commands for predefined and custom steps. Those commands are supposed to be executed by the shell/command line. According to the documentation and the source code of ExternalProject, directory tokens are mainly supposed to be replaced in commands. It is my understanding, that it is a bug, if CMake isn't able to assemble these commands correctly. This would include usage of the correct path style of the OS for shell/command line commands. As directory tokens are replaced internally right before a shell/command line command is assembled, I can't see why this would be kind of API-breaking. You cannot interfere in your CMake code with these internal replacements. Therefore I would still prefer my solution as it is pretty simple without adding even more features to ExternalProject and in my opinion without breaking code in the wild. It is a true bug fix instead of a feature request for working directories, which is a different topic that just coincidentally arised because of my specific example I guess. The features you described wouldn't fix the actual bug. As you were not sure if my approach would even fix my problems: It does of course and this is what I am currently doing and what I tested extensively before creating the patch. :) Regarding your quote from the add_custom_command documentation I can tell you
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
I still disagree on the point that CMake shouldn't be fixed because of possibly erroneous external scripts that are not able to handle paths which were - again possibly - passed to them as parameters in the style of the platform they were written for. This is very hypothetical in my opinion. We should also bring to mind here that CMake as seen as an entity as well as CMake scripts wouldn't be affected by the fix at all regarding backwards compatibility. How can these obviously rare cases - if existent at all - lie in the scope of responsibility of CMake? And how could one justify that everyone (else?) have to continue working around this bug? Please be aware that I am not asking that EVERYONE should solve a problem which I have encountered and that this is far from it just works. In fact, James Johnston even ran into trouble with this bug yesterday coincidentally and I know at least of colleagues working around this issue for years now. It just doesn't work and there is no way that one could know this without burning lifetime figuring out what's going on and how to work around. Referring to the latest email of James Johnston and your advice not to use directory tokens, you see, that this, on one hand, is not a good option sometimes (and this would ironically create potential to break backwards compatibility as one would need to maintain script that kind of mimics CMake internals). On the other hand it is a little strange as it is rather equivalent to to say not to use the feature for what is solely intended to at all. Obviously, something has to be done. The question seems to be: What should be done if you still want to keep the erroneous behavior? - Point out the issue in the documentation (and wish good luck to CMake users for future upgrades of CMake as they are potentially mimicking potentially changing internals instead)? :) - Adding a CMake policy? - Adding a new *single* binary option to ExternalProject_* functions in the style of the BUILD_IN_SOURCE 1 option, e.g. NATIVE_DIR_TOKENS? - Something else? I would still prefer to fix the bug directly as the only counter argument until now was about (loosely coupled) backwards compatibility and I hope I could show above that actually the reverse is true, and there is an on-going (loosely coupled) backwards compatibility issue. Stefan Von: cmake-developers [cmake-developers-boun...@cmake.org] im Auftrag von David Cole via cmake-developers [cmake-developers@cmake.org] Gesendet: Donnerstag, 20. August 2015 23:20 An: James Johnston Cc: cmake-developers@cmake.org Betreff: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens It's exactly what I am concerned about: You're asking to change the behavior of something for EVERYONE to solve a problem which you have encountered. If you change it the way you have proposed, you will cause problems for others. It has worked the way it is now since ExternalProject_Add was introduced in CMake 2.8. Changing it unconditionally the way you propose is simply not feasible for backwards compatibility. I think commands that take native paths ought NOT to use the *_DIR replacement values, and instead, ought to pass in variables that contain the native paths in the first place. David C. On Thu, Aug 20, 2015 at 2:58 PM, James Johnston johnstonj.pub...@codenest.com wrote: Hi, Funny you are mailing the list about this, since I just ran into this same issue today building something totally different from Boost... In this case it's a build tool that thinks the /U in C:/Users is a new command line argument, that isn't recognized - and then the subsequent s also ends up unrecognized... and it all fails... And it has nothing to do with the working directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible workaround for me. I think the issue with globally making this change to the existing tokens is that there could be some external tool/program that is EXPECTING to get CMake paths, not native paths. Who knows? I am guessing that is what David Cole was concerned about. Maybe the right answer is to introduce some NEW tokens while leaving the behavior of the old ones unchanged. E.g. BINARY_DIR_NATIVE etc. It would be good if the patch updates the documentation of ExternalProject and clearly states the path format of BINARY_DIR vs BINARY_DIR_NATIVE. Then the user can pick whichever one suits them best, depending on the tool being invoked. Furthermore, sometimes BINARY_DIR_NATIVE still needs to be replaced with a CMake path, not native path. For example, if the token is being found in a property like WORKING_DIRECTORY that eventually gets passed to add_custom_command(WORKING_DIRECTORY) then I'm guessing still has to be a CMake path. I am guessing this is what David Cole was also concerned about. I still think your original method of building Boost is a bit unusual
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
Hi,
Funny you are mailing the list about this, since I just ran into this same
issue today building something totally different from Boost... In this case
it's a build tool that thinks the /U in C:/Users is a new command line
argument, that isn't recognized - and then the subsequent s also ends up
unrecognized... and it all fails... And it has nothing to do with the working
directory, so _Add_Step(WORKING_DIRECTORY) isn't a possible workaround for me.
I think the issue with globally making this change to the existing tokens is
that there could be some external tool/program that is EXPECTING to get CMake
paths, not native paths. Who knows? I am guessing that is what David Cole was
concerned about.
Maybe the right answer is to introduce some NEW tokens while leaving the
behavior of the old ones unchanged. E.g. BINARY_DIR_NATIVE etc. It would be
good if the patch updates the documentation of ExternalProject and clearly
states the path format of BINARY_DIR vs BINARY_DIR_NATIVE. Then the user
can pick whichever one suits them best, depending on the tool being invoked.
Furthermore, sometimes BINARY_DIR_NATIVE still needs to be replaced with a
CMake path, not native path. For example, if the token is being found in a
property like WORKING_DIRECTORY that eventually gets passed to
add_custom_command(WORKING_DIRECTORY) then I'm guessing still has to be a CMake
path. I am guessing this is what David Cole was also concerned about.
I still think your original method of building Boost is a bit unusual and would
be better served by _Add_Step with a custom working directory - because that's
the publicly documented/standard way of changing the working directory, but
that is up to you. :)
Best regards,
James Johnston
On Thu, 20 Aug 2015 14:37:08 + Stefan Kislinskiy
s.kislins...@dkfz-heidelberg.de wrote
Hi,
thank you for our suggestions. I am aware that I can solve my example
differently and that it might look not directly connected the proposal, but
well, it is an example just to show a single case and why it matters. :) I
did not want to discuss the example itself. Working around here would just
resolve a symptom.
My point is the overall problem that would persist: A big part of
ExternalProject is to issue commands for predefined and custom steps. Those
commands are supposed to be executed by the shell/command line. According to
the documentation and the source code of ExternalProject, directory tokens
are mainly supposed to be replaced in commands. It is my understanding, that
it is a bug, if CMake isn't able to assemble these commands correctly. This
would include usage of the correct path style of the OS for shell/command
line commands. As directory tokens are replaced internally right before a
shell/command line command is assembled, I can't see why this would be kind
of API-breaking. You cannot interfere in your CMake code with these
internal replacements.
Therefore I would still prefer my solution as it is pretty simple without
adding even more features to ExternalProject and in my opinion without
breaking code in the wild. It is a true bug fix instead of a feature request
for working directories, which is a different topic that just coincidentally
arised because of my specific example I guess. The features you described
wouldn't fix the actual bug.
As you were not sure if my approach would even fix my problems: It does of
course and this is what I am currently doing and what I tested extensively
before creating the patch. :) Regarding your quote from the
add_custom_command documentation I can tell you that this is how things are
currently done in ExternalProject and always were as far as I know, for
example (from ExternalProject.cmake):
add_custom_command(
OUTPUT ${stamp_file}
BYPRODUCTS ${byproducts}
COMMENT ${comment}
COMMAND ${command}
COMMAND ${touch}
DEPENDS ${depends}
WORKING_DIRECTORY ${work_dir}
VERBATIM
)
Best regards,
Stefan
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org] On Behalf
Of James Johnston
Sent: Donnerstag, 20. August 2015 15:37
To: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
On Behalf Of Kislinskiy, Stefan
Sent: Thursday, August 20, 2015 09:02
To: David Cole
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
Hi David,
Example excerpt (it is not possible to change the working directory
for
the
CONFIGURE_COMMAND as it is fixed to the BUILD_DIR, which might
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
Hi,
thank you for our suggestions. I am aware that I can solve my example
differently and that it might look not directly connected the proposal, but
well, it is an example just to show a single case and why it matters. :) I did
not want to discuss the example itself. Working around here would just resolve
a symptom.
My point is the overall problem that would persist: A big part of
ExternalProject is to issue commands for predefined and custom steps. Those
commands are supposed to be executed by the shell/command line. According to
the documentation and the source code of ExternalProject, directory tokens are
mainly supposed to be replaced in commands. It is my understanding, that it is
a bug, if CMake isn't able to assemble these commands correctly. This would
include usage of the correct path style of the OS for shell/command line
commands. As directory tokens are replaced internally right before a
shell/command line command is assembled, I can't see why this would be kind of
API-breaking. You cannot interfere in your CMake code with these internal
replacements.
Therefore I would still prefer my solution as it is pretty simple without
adding even more features to ExternalProject and in my opinion without breaking
code in the wild. It is a true bug fix instead of a feature request for working
directories, which is a different topic that just coincidentally arised because
of my specific example I guess. The features you described wouldn't fix the
actual bug.
As you were not sure if my approach would even fix my problems: It does of
course and this is what I am currently doing and what I tested extensively
before creating the patch. :) Regarding your quote from the add_custom_command
documentation I can tell you that this is how things are currently done in
ExternalProject and always were as far as I know, for example (from
ExternalProject.cmake):
add_custom_command(
OUTPUT ${stamp_file}
BYPRODUCTS ${byproducts}
COMMENT ${comment}
COMMAND ${command}
COMMAND ${touch}
DEPENDS ${depends}
WORKING_DIRECTORY ${work_dir}
VERBATIM
)
Best regards,
Stefan
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org] On Behalf Of
James Johnston
Sent: Donnerstag, 20. August 2015 15:37
To: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as substitute
for directory tokens
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
On Behalf Of Kislinskiy, Stefan
Sent: Thursday, August 20, 2015 09:02
To: David Cole
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
Hi David,
Example excerpt (it is not possible to change the working directory
for
the
CONFIGURE_COMMAND as it is fixed to the BUILD_DIR, which might not be
sufficient):
This doesn't really directly have to do with your proposal, but what if an
option was added to change the working dir of the CONFIGURE_COMMAND? E.g.
WORKING_DIRECTORY_CONFIGURE. And suppose you'd have it recognize the various
tags like SOURCE_DIR, etc. This might be useful to add to other steps as
well, and would be more portable than your solution which is using
cmd.exe-specific commands. You'd want to audit for any resulting breakage
(e.g. does ExternalProject make assumptions that the working directory of
CONFIGURE is always the binary dir? - e.g. a relative path being used
somewhere. And probably only allow specification of
WORKING_DIRECTORY_CONFIGURE if a CONFIGURE_COMMAND was also specified, as the
built-in commands certainly assume the default working dir.)
In your situation though, I'm not sure it's strictly needed. From your sample,
it looks like you're building boost. In your case what if you:
* Use ExternalProject_Add_Step to bootstrap. You can specify a
WORKING_DIRECTORY here. Note one problem: you can't do out of source build of
b2, which breaks user expectations.
* Then use ExternalProject_Add_Step to build Boost.
Yes, using _Add_Step is somewhat of a workaround, but in this case, I've found
it wasn't much of a burden at all. In fact the only case I can think of where
it WOULD be a burden would be if the configure step is CMake. But then you
wouldn't need to change the working directory; changing it would break CMake.
In practice nobody will want to change WORKING_DIRECTORY unless it's a custom
command and then it's easy to use _Add_Step anyway.
That said, it might still be considered a little undesired and so maybe my
proposal above would be a better way to handle it.
Corrections from maintainers and others on the above commentary are welcome...
set(bootstrap_cmd SOURCE_DIR/bootstrap${shell_ext}
${bootstrap_toolset})
if(WIN32)
set(bootstrap_cmd pushd SOURCE_DIR COMMAND ${bootstrap_cmd}
COMMAND popd)
endif()
ExternalProject_Add(Boost
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
On Behalf Of Kislinskiy, Stefan
Sent: Thursday, August 20, 2015 09:02
To: David Cole
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
Hi David,
Example excerpt (it is not possible to change the working directory for
the
CONFIGURE_COMMAND as it is fixed to the BUILD_DIR, which might not be
sufficient):
This doesn't really directly have to do with your proposal, but what if an
option was added to change the working dir of the CONFIGURE_COMMAND? E.g.
WORKING_DIRECTORY_CONFIGURE. And suppose you'd have it recognize the
various tags like SOURCE_DIR, etc. This might be useful to add to other
steps as well, and would be more portable than your solution which is using
cmd.exe-specific commands. You'd want to audit for any resulting breakage
(e.g. does ExternalProject make assumptions that the working directory of
CONFIGURE is always the binary dir? - e.g. a relative path being used
somewhere. And probably only allow specification of
WORKING_DIRECTORY_CONFIGURE if a CONFIGURE_COMMAND was also specified, as
the built-in commands certainly assume the default working dir.)
In your situation though, I'm not sure it's strictly needed. From your
sample, it looks like you're building boost. In your case what if you:
* Use ExternalProject_Add_Step to bootstrap. You can specify a
WORKING_DIRECTORY here. Note one problem: you can't do out of source build
of b2, which breaks user expectations.
* Then use ExternalProject_Add_Step to build Boost.
Yes, using _Add_Step is somewhat of a workaround, but in this case, I've
found it wasn't much of a burden at all. In fact the only case I can think
of where it WOULD be a burden would be if the configure step is CMake. But
then you wouldn't need to change the working directory; changing it would
break CMake. In practice nobody will want to change WORKING_DIRECTORY
unless it's a custom command and then it's easy to use _Add_Step anyway.
That said, it might still be considered a little undesired and so maybe my
proposal above would be a better way to handle it.
Corrections from maintainers and others on the above commentary are
welcome...
set(bootstrap_cmd SOURCE_DIR/bootstrap${shell_ext}
${bootstrap_toolset})
if(WIN32)
set(bootstrap_cmd pushd SOURCE_DIR COMMAND ${bootstrap_cmd}
COMMAND popd)
endif()
ExternalProject_Add(Boost
...
CONFIGURE_COMMAND ${bootstrap_cmd}
...
)
From add_custom_command: If more than one COMMAND is specified they will
be executed in order, but not necessarily composed into a stateful shell or
batch script.
So I am not sure your approach will work for you even if you fix the issue
with path slashes.
James
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
. August 2015 15:37
To: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
-Original Message-
From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
On Behalf Of Kislinskiy, Stefan
Sent: Thursday, August 20, 2015 09:02
To: David Cole
Cc: cmake-developers@cmake.org
Subject: Re: [cmake-developers] ExternalProject: Use native paths as
substitute for directory tokens
Hi David,
Example excerpt (it is not possible to change the working directory
for
the
CONFIGURE_COMMAND as it is fixed to the BUILD_DIR, which might not be
sufficient):
This doesn't really directly have to do with your proposal, but what if an
option was added to change the working dir of the CONFIGURE_COMMAND? E.g.
WORKING_DIRECTORY_CONFIGURE. And suppose you'd have it recognize the
various tags like SOURCE_DIR, etc. This might be useful to add to other
steps as well, and would be more portable than your solution which is using
cmd.exe-specific commands. You'd want to audit for any resulting breakage
(e.g. does ExternalProject make assumptions that the working directory of
CONFIGURE is always the binary dir? - e.g. a relative path being used
somewhere. And probably only allow specification of
WORKING_DIRECTORY_CONFIGURE if a CONFIGURE_COMMAND was also specified, as the
built-in commands certainly assume the default working dir.)
In your situation though, I'm not sure it's strictly needed. From your
sample, it looks like you're building boost. In your case what if you:
* Use ExternalProject_Add_Step to bootstrap. You can specify a
WORKING_DIRECTORY here. Note one problem: you can't do out of source build
of b2, which breaks user expectations.
* Then use ExternalProject_Add_Step to build Boost.
Yes, using _Add_Step is somewhat of a workaround, but in this case, I've
found it wasn't much of a burden at all. In fact the only case I can think
of where it WOULD be a burden would be if the configure step is CMake. But
then you wouldn't need to change the working directory; changing it would
break CMake. In practice nobody will want to change WORKING_DIRECTORY unless
it's a custom command and then it's easy to use _Add_Step anyway.
That said, it might still be considered a little undesired and so maybe my
proposal above would be a better way to handle it.
Corrections from maintainers and others on the above commentary are
welcome...
set(bootstrap_cmd SOURCE_DIR/bootstrap${shell_ext}
${bootstrap_toolset})
if(WIN32)
set(bootstrap_cmd pushd SOURCE_DIR COMMAND ${bootstrap_cmd}
COMMAND popd)
endif()
ExternalProject_Add(Boost
...
CONFIGURE_COMMAND ${bootstrap_cmd}
...
)
From add_custom_command: If more than one COMMAND is specified they will
be executed in order, but not necessarily composed into a stateful shell or
batch script.
So I am not sure your approach will work for you even if you fix the issue
with path slashes.
James
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
-Original Message- From: cmake-developers [mailto:cmake-developers-boun...@cmake.org] On Behalf Of David Cole via cmake-developers Sent: Thursday, August 20, 2015 21:21 To: James Johnston Cc: cmake-developers@cmake.org Subject: Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens It's exactly what I am concerned about: You're asking to change the behavior of something for EVERYONE to solve a problem which you have encountered. If you change it the way you have proposed, you will cause problems for others. It has worked the way it is now since ExternalProject_Add was introduced in CMake 2.8. Changing it unconditionally the way you propose is simply not feasible for backwards compatibility. I think commands that take native paths ought NOT to use the *_DIR replacement values, and instead, ought to pass in variables that contain the native paths in the first place. Right, agreed that the original *_DIR behavior would need to remain unchanged. But how would you know what the binary directory of the external project is, before calling ExternalProject_Add? It's too early to call ExternalProject_Get_Property(target BINARY_DIR). I assume that is why BINARY_DIR and friends exist, as the location will be affected via various ExternalProject parameters. From the doc: The *_DIR options specify directories for the project, with default directories computed as follows snip Quite a set of rules follow and if one is trying to use the binary directory in the path to some build tool, it's handy to put BINARY_DIR in the command. But if you need a native path for your build tool, BINARY_DIR doesn't work as Stefan Kislinskiy noted, which is why I suggested making new tokens might be a viable alternative, e.g. BINARY_DIR_NATIVE? Best regards, James Johnston -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
[cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
Hi,
I would like to contribute a fix regarding directory tokens (e.g. SOURCE_DIR)
in the ExternalProject module. These tokens are replaced by their matching
target property value _EP_${dir}. However, the path substitutes are not
converted to native path style which makes it impossible to use directory
tokens in some Windows commands like pushd, as these commands are not able to
handle forward slashes. I attached a tiny patch which simply calls
file(TO_NATIVE_PATH) before replacing directory tokens.
Best regards,
Stefan Kislinskiy
external-project-native-dir-tokens.patch
Description: external-project-native-dir-tokens.patch
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] ExternalProject: Use native paths as substitute for directory tokens
But some existing replacements require forward slashes, we can't just to
native path everything blindly. I think you will have to special case the
bits which do need native paths.
On Wednesday, August 19, 2015, Kislinskiy, Stefan
s.kislins...@dkfz-heidelberg.de wrote:
Hi,
I would like to contribute a fix regarding directory tokens (e.g.
SOURCE_DIR) in the ExternalProject module. These tokens are replaced by
their matching target property value _EP_${dir}. However, the path
substitutes are not converted to native path style which makes it
impossible to use directory tokens in some Windows commands like pushd, as
these commands are not able to handle forward slashes. I attached a tiny
patch which simply calls file(TO_NATIVE_PATH) before replacing directory
tokens.
Best regards,
Stefan Kislinskiy
--
Powered by www.kitware.com
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more
information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers
