Re: [CMake] How to deal with generated source files (w/ dependency tracking) for globs

[Craig Scott]

It's not quite your situation, but it may have some common elements and the
general idea may get you thinking about what might work for you. This
question/answer I asked this list about last week:

http://stackoverflow.com/q/36084785/1938798

We've been using this approach with success on all the platforms listed
there. In your case, you may want to consider getting your tool that
converts from Nim to C to do that off to the side and only replace the
existing generated C sources with the new ones if they are different.
That's what our particular tool in the above linked article does and it
works rather well for us all as part of one single build. I haven't tried
building within Xcode directly, but we do have scripted builds for iOS
using the Xcode generator which are working.




On Thu, Apr 14, 2016 at 8:40 PM, Eric Wing  wrote:

> On 4/6/16, iosif neitzke  wrote:
> > I think it depends on when you want the output files from Nim
> > generated and which files are the most frequently developed.
> > If it is usually a one-time generation per clean development session,
> > the simplest case, where the *.NIM source files are not the files
> > most likely to be changed, I would think execute_process should
> > work okay?
> >
>
> Thanks for the responses. In this case, the common use case would be
> writing/changing the NIM files, which in turn require the C files to
> be rewritten. (The users are writing in NIM, and C generation to them
> is just an implementation detail and means to an end.)
>
> It sounds like this is going to be a pain.
>
> I appreciate everybody's responses on this.
>
> Thanks,
> Eric
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>



-- 
Craig Scott
Melbourne, Australia
http://crascit.com
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] How to deal with generated source files (w/ dependency tracking) for globs

[Eric Wing]

On 4/6/16, iosif neitzke  wrote:
> I think it depends on when you want the output files from Nim
> generated and which files are the most frequently developed.
> If it is usually a one-time generation per clean development session,
> the simplest case, where the *.NIM source files are not the files
> most likely to be changed, I would think execute_process should
> work okay?
>

Thanks for the responses. In this case, the common use case would be
writing/changing the NIM files, which in turn require the C files to
be rewritten. (The users are writing in NIM, and C generation to them
is just an implementation detail and means to an end.)

It sounds like this is going to be a pain.

I appreciate everybody's responses on this.

Thanks,
Eric
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] How to deal with generated source files (w/ dependency tracking) for globs

[iosif neitzke]

I think it depends on when you want the output files from Nim
generated and which files are the most frequently developed.
If it is usually a one-time generation per clean development session,
the simplest case, where the *.NIM source files are not the files
most likely to be changed, I would think execute_process should
work okay?

cmake_minimum_required( VERSION 2.8.11 )

project( GenerateSomeFilesAndBuildThem C )

execute_process( COMMAND touch a.c a.h b.c b.h c.c c.h
   COMMAND echo "Generating some C files using NIM..."
   WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} )

file( GLOB SRC ${CMAKE_CURRENT_BINARY_DIR}/*.c
   ${CMAKE_CURRENT_BINARY_DIR}/*.h )

add_executable( generated main.c ${SRC} )

target_include_directories( generated PRIVATE ${CMAKE_CURRENT_BINARY_DIR} )

...

The C compiler identification is GNU 5.2.1
Check for working C compiler: /usr/bin/cc
Check for working C compiler: /usr/bin/cc -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Detecting C compile features
Detecting C compile features - done
Generating some C files using NIM...

Configuring done
Generating done

...

make
Scanning dependencies of target generated
[ 25%] Building C object CMakeFiles/generated.dir/main.c.o
[ 50%] Building C object CMakeFiles/generated.dir/c.c.o
[ 75%] Building C object CMakeFiles/generated.dir/a.c.o
[100%] Building C object CMakeFiles/generated.dir/b.c.o
Linking C executable generated
[100%] Built target generated

On Wed, Apr 6, 2016 at 8:55 AM, Nicholas Braden
 wrote:
> Okay, so I tried my own suggesting and found that it doesn't work at
> all (it's more complicated than I thought). However I want to ask you
> about your logic here - you say that when the configure step runs, it
> cancels XCode's build and so the user has to run the build a second
> time. If we use a script to generate a file or just touch one of the
> cmake files for every build, that means that every build will be
> cancelled and XCode will never move on to actually compile anything.
> The only way to actually let a build not be cancelled is either to run
> `cmake --build .` from the command line instead of using the IDE, or
> to not modify any CMake file, which requires some pretty elaborate
> scripting to accomplish. Even just touching the modified date of a
> CMake file will cause the configure step to run. At this point I think
> you should just tell your users to manually run the configure step
> whenever they add or remove a nim source file, because either way it
> will be a two-click process. There is no way I see to have the magic
> single-click build you want given XCode's limitations. But if you ever
> do find a way I'd be interested to hear.
>
> (Though, I don't know how XCode treats External Projects - maybe you
> could get it to work that way, but it'd be ugly)
>
> On Wed, Apr 6, 2016 at 6:28 AM, Eric Wing  wrote:
>> On 4/4/16, Nicholas Braden  wrote:
>>> I haven't tested this myself, but instead of using a glob, you could
>>> have a build step that generates a CMake script file with the list of
>>> generated files in it (e.g. via execute_process to run another CMake
>>> script whose only job is to generate the CMake script inception-style)
>>> and just include the generated file - CMake will see that the CMake
>>> file has been modified during every build but will still handle source
>>> file changes correctly. Worth a shot.
>>>
>>
>> Thanks for the reply. So if I understand this correctly, this is what
>> the workflow is going to feel like.
>>
>> - User adds their Nim file to the CMakeLists.txt
>> - CMake detects a change and a bootstrap generate/update takes place,
>> and my stuff re-runs the Nim compiler.
>> - Also during this phase, I need to generate another CMake script
>> which contains the list of files the Nim compiler output.
>> - Since this other script is also under CMake change tracking, this
>> will trigger another CMake change detection and
>> bootstrap/generate/update.
>> - Now the generated build system has the updated list of generated .c
>> files to compile and will finally compile it.
>>
>> Does this sound right?
>>
>> My major concern is that the two bootstrap phases kind of break the
>> flow. What I mean by that is that in Xcode for example, when a
>> bootstrap/generate/update happens, it actually causes your Build/Run
>> (run button) action to abort midway through. You sit there expecting
>> it to build, but CMake's regeneration was the only thing that
>> happened. You must hit the Run button again to actually build/run.
>> Kind of annoying and also really confusing for people who don't
>> understand this because it isn't obvious why this happened.
>>
>> But in the above example, there are now two bootstrap phases. So
>> assuming this works, I think the user will have to hit the Run button
>> at least 3 times before something actually builds/runs. I think this
>> is going to feel real 

Re: [CMake] How to deal with generated source files (w/ dependency tracking) for globs

[Nicholas Braden]

Okay, so I tried my own suggesting and found that it doesn't work at
all (it's more complicated than I thought). However I want to ask you
about your logic here - you say that when the configure step runs, it
cancels XCode's build and so the user has to run the build a second
time. If we use a script to generate a file or just touch one of the
cmake files for every build, that means that every build will be
cancelled and XCode will never move on to actually compile anything.
The only way to actually let a build not be cancelled is either to run
`cmake --build .` from the command line instead of using the IDE, or
to not modify any CMake file, which requires some pretty elaborate
scripting to accomplish. Even just touching the modified date of a
CMake file will cause the configure step to run. At this point I think
you should just tell your users to manually run the configure step
whenever they add or remove a nim source file, because either way it
will be a two-click process. There is no way I see to have the magic
single-click build you want given XCode's limitations. But if you ever
do find a way I'd be interested to hear.

(Though, I don't know how XCode treats External Projects - maybe you
could get it to work that way, but it'd be ugly)

On Wed, Apr 6, 2016 at 6:28 AM, Eric Wing  wrote:
> On 4/4/16, Nicholas Braden  wrote:
>> I haven't tested this myself, but instead of using a glob, you could
>> have a build step that generates a CMake script file with the list of
>> generated files in it (e.g. via execute_process to run another CMake
>> script whose only job is to generate the CMake script inception-style)
>> and just include the generated file - CMake will see that the CMake
>> file has been modified during every build but will still handle source
>> file changes correctly. Worth a shot.
>>
>
> Thanks for the reply. So if I understand this correctly, this is what
> the workflow is going to feel like.
>
> - User adds their Nim file to the CMakeLists.txt
> - CMake detects a change and a bootstrap generate/update takes place,
> and my stuff re-runs the Nim compiler.
> - Also during this phase, I need to generate another CMake script
> which contains the list of files the Nim compiler output.
> - Since this other script is also under CMake change tracking, this
> will trigger another CMake change detection and
> bootstrap/generate/update.
> - Now the generated build system has the updated list of generated .c
> files to compile and will finally compile it.
>
> Does this sound right?
>
> My major concern is that the two bootstrap phases kind of break the
> flow. What I mean by that is that in Xcode for example, when a
> bootstrap/generate/update happens, it actually causes your Build/Run
> (run button) action to abort midway through. You sit there expecting
> it to build, but CMake's regeneration was the only thing that
> happened. You must hit the Run button again to actually build/run.
> Kind of annoying and also really confusing for people who don't
> understand this because it isn't obvious why this happened.
>
> But in the above example, there are now two bootstrap phases. So
> assuming this works, I think the user will have to hit the Run button
> at least 3 times before something actually builds/runs. I think this
> is going to feel real broken and feel like dumb luck something
> actually built.
>
> So I was hoping to find a one-step bootstrap solution instead of two-step.
>
> Thanks,
> Eric
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] How to deal with generated source files (w/ dependency tracking) for globs

[Eric Wing]

On 4/4/16, Nicholas Braden  wrote:
> I haven't tested this myself, but instead of using a glob, you could
> have a build step that generates a CMake script file with the list of
> generated files in it (e.g. via execute_process to run another CMake
> script whose only job is to generate the CMake script inception-style)
> and just include the generated file - CMake will see that the CMake
> file has been modified during every build but will still handle source
> file changes correctly. Worth a shot.
>

Thanks for the reply. So if I understand this correctly, this is what
the workflow is going to feel like.

- User adds their Nim file to the CMakeLists.txt
- CMake detects a change and a bootstrap generate/update takes place,
and my stuff re-runs the Nim compiler.
- Also during this phase, I need to generate another CMake script
which contains the list of files the Nim compiler output.
- Since this other script is also under CMake change tracking, this
will trigger another CMake change detection and
bootstrap/generate/update.
- Now the generated build system has the updated list of generated .c
files to compile and will finally compile it.

Does this sound right?

My major concern is that the two bootstrap phases kind of break the
flow. What I mean by that is that in Xcode for example, when a
bootstrap/generate/update happens, it actually causes your Build/Run
(run button) action to abort midway through. You sit there expecting
it to build, but CMake's regeneration was the only thing that
happened. You must hit the Run button again to actually build/run.
Kind of annoying and also really confusing for people who don't
understand this because it isn't obvious why this happened.

But in the above example, there are now two bootstrap phases. So
assuming this works, I think the user will have to hit the Run button
at least 3 times before something actually builds/runs. I think this
is going to feel real broken and feel like dumb luck something
actually built.

So I was hoping to find a one-step bootstrap solution instead of two-step.

Thanks,
Eric
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] How to deal with generated source files (w/ dependency tracking) for globs

[Nicholas Braden]

I haven't tested this myself, but instead of using a glob, you could
have a build step that generates a CMake script file with the list of
generated files in it (e.g. via execute_process to run another CMake
script whose only job is to generate the CMake script inception-style)
and just include the generated file - CMake will see that the CMake
file has been modified during every build but will still handle source
file changes correctly. Worth a shot.

On Mon, Apr 4, 2016 at 6:36 PM, Eric Wing  wrote:
> Hi, this is kind of a side problem that was asked of me. I'm not sure
> how many hoops I actually want to jump through to solve this. But,
> here's the situation.
>
> I have a really nice, traditional C based project in CMake. All my
> dependency tracking works really well for cross-platform and IDE
> integration and even have resources/bundling/codesigning working
> nicely in the single build step.
>
> Now somebody throws me a curve ball and wants to use the Nim language.
> The specifics of the Nim language itself is not that important to this
> topic, but the key piece of information is that the Nim compiler can
> generate C files as output.
>
> So the question is, how can I invoke Nim to compile a list of
> pre-known Nim source files, and then slurp in the generated output C
> files and build them as part of my normal CMake build process.
>
> The curve ball is this: I don't know all the names or how many files
> the Nim compiler generates. If I know all the names beforehand, I
> think I know how to deal with this situation. But instead, I get a
> directory of C files. The names seem to be dependent on which library
> features were used and also what the user's source files were named. I
> suspect the standard library could theoretically change too so future
> versions of Nim may have different files.
>
> I know file globs with CMake don't auto-track changes and I think the
> problem gets more trickier because of the indirection to invoke Nim.
> (i.e. When I first glob the files, the directory is empty, then the
> Nim compiler is invoked later and the files appear, but it is too late
> for CMake to see without a regeneration).
>
> Anyway, I was wondering what the best approach is.
>
> Thanks,
> Eric
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

[CMake] How to deal with generated source files (w/ dependency tracking) for globs

[Eric Wing]

Hi, this is kind of a side problem that was asked of me. I'm not sure
how many hoops I actually want to jump through to solve this. But,
here's the situation.

I have a really nice, traditional C based project in CMake. All my
dependency tracking works really well for cross-platform and IDE
integration and even have resources/bundling/codesigning working
nicely in the single build step.

Now somebody throws me a curve ball and wants to use the Nim language.
The specifics of the Nim language itself is not that important to this
topic, but the key piece of information is that the Nim compiler can
generate C files as output.

So the question is, how can I invoke Nim to compile a list of
pre-known Nim source files, and then slurp in the generated output C
files and build them as part of my normal CMake build process.

The curve ball is this: I don't know all the names or how many files
the Nim compiler generates. If I know all the names beforehand, I
think I know how to deal with this situation. But instead, I get a
directory of C files. The names seem to be dependent on which library
features were used and also what the user's source files were named. I
suspect the standard library could theoretically change too so future
versions of Nim may have different files.

I know file globs with CMake don't auto-track changes and I think the
problem gets more trickier because of the indirection to invoke Nim.
(i.e. When I first glob the files, the directory is empty, then the
Nim compiler is invoked later and the files appear, but it is too late
for CMake to see without a regeneration).

Anyway, I was wondering what the best approach is.

Thanks,
Eric
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake