Re: [cmake-developers] Compile targets affected by changeset

[Florent Castelli]

Depending on your setup and the amount of build machines, you may have 
even better results with sccache from Mozilla. It's a new implementation 
similar to ccache with an optional distributed cache in S3 or Redis.


The downside is if you have long running tests in the list, it won't 
help you much (unless you have a way to checksum the test programs and 
skip them maybe?). But still, it's always a good thing to rerun them as 
you may have some unstable tests and more runs means more confidence.


/Florent

On 22/05/2017 23:06, Craig Scott wrote:
I highly recommend Florent's suggestion. We use ccache on our CI 
system and for local development. We've stopped worrying about how 
long builds take now, since only files that are changed or that rely 
on things that changed contribute any meaningful amount to the build 
time. It also works for Make, Ninja and Xcode, so it's more flexible 
than relying on some feature of Make and it is also pretty easy to set 
up. You can set it up system wide, or you can take the approach 
discussed in this article 
 for systems 
where you don't have access to set up ccache globally.



On Tue, May 23, 2017 at 4:54 AM, Florent Castelli 
mailto:florent.caste...@gmail.com>> wrote:




On 22 May 2017, at 20:07, Robert Patterson via cmake-developers
mailto:cmake-developers@cmake.org>>
wrote:

We understand that CMake and make already can rebuild targets
which depend on changed files, and this behavior works exactly as
expected for us. Our issue is not that make is rebuilding targets
that it shouldn't. We would like this 'compile targets which
changed' behavior to work from a clean state, not just for
subsequent rebuilds. That is, from the first time cmake / make is
invoked, only the targets that depend on the given set of files
will be built, and no more.

When a developer submits a changeset to our continuous
integration tool, a fresh copy of the repository is checked out
by the worker at the commit of the changeset. We can access the
set of changed files from the commit data. Currently, cmake and
make correctly identify that the system is in a clean state and
that everything should be built if the current top-level target
is specified. This is the point where we would like our behavior
to differ. We want the rebuild changed targets behavior as if
everything were already built, but from this clean state.

Once we have determined the targets to build, we could simply
specify these targets as goals to make in the command line, but
we are hampered by the following limitation of make.

We are using GNU make 3.82. Invoking 'make -j target1 target2
target3' on the command line specifies the targets as goals to
make. If several goals are specified, 'make' processes each of
them in turn, in the order you name them.
https://www.gnu.org/software/make/manual/html_node/Goals.html


Ninja does not currently work for our project.


It would be interesting to fix Ninja for your project then. Do you
know what isn’t currently working?
Also, Ninja has some APIs to expose all the dependency graph,
which you could probably query somehow to find out which target
you need to rebuild.

Another possibility, albeit a bit different, could be to use a
compilation cache to just rebuild everything but much faster, to
the point that it might not be relevant anymore.

/Florent




On May 19, 2017, at 2:32 AM, Simon Richter
mailto:simon.rich...@hogyros.de>> wrote:

Hi,

On 18.05.2017 23:48, Robert Patterson via cmake-developers wrote:


My company has a large, predominately C++ codebase, with
hundreds of
targets, both for product and unit tests.  In an effort to
improve the
compile and test time for developers, which utilizes a continuous
integration infrastructure, it is desirable to compile only the
targets
that are affected by developer's change sets.


Erm, it should already work this way. If Make rebuilds a target
that it
shouldn't, the first step would be investigating why it thinks the
target needs to be rebuilt.

Dependency tracking is one of the oldest problems, and cmake
should use
an appropriate solution for the actual build system you use. The
approach used for Make is a bit more conservative than you would
require
for GNU Make, but should nevertheless still work.

http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/


is a good primer on dependency list generation with Make -- CMake
generates rules that are similar to these.


'make' has a limitation where if 'make target1 target2 target3' is
invoked, target1, target2, and 

Re: [cmake-developers] Compile targets affected by changeset

[Craig Scott]

I highly recommend Florent's suggestion. We use ccache on our CI system and
for local development. We've stopped worrying about how long builds take
now, since only files that are changed or that rely on things that changed
contribute any meaningful amount to the build time. It also works for Make,
Ninja and Xcode, so it's more flexible than relying on some feature of Make
and it is also pretty easy to set up. You can set it up system wide, or you
can take the approach discussed in this article
 for systems where
you don't have access to set up ccache globally.


On Tue, May 23, 2017 at 4:54 AM, Florent Castelli <
florent.caste...@gmail.com> wrote:

>
> On 22 May 2017, at 20:07, Robert Patterson via cmake-developers <
> cmake-developers@cmake.org> wrote:
>
> We understand that CMake and make already can rebuild targets which depend
> on changed files, and this behavior works exactly as expected for us. Our
> issue is not that make is rebuilding targets that it shouldn't. We would
> like this 'compile targets which changed' behavior to work from a clean
> state, not just for subsequent rebuilds. That is, from the first time cmake
> / make is invoked, only the targets that depend on the given set of files
> will be built, and no more.
>
> When a developer submits a changeset to our continuous integration tool, a
> fresh copy of the repository is checked out by the worker at the commit of
> the changeset. We can access the set of changed files from the commit data.
> Currently, cmake and make correctly identify that the system is in a clean
> state and that everything should be built if the current top-level target
> is specified. This is the point where we would like our behavior to differ.
> We want the rebuild changed targets behavior as if everything were already
> built, but from this clean state.
>
> Once we have determined the targets to build, we could simply specify
> these targets as goals to make in the command line, but we are hampered by
> the following limitation of make.
>
> We are using GNU make 3.82. Invoking 'make -j target1 target2 target3' on
> the command line specifies the targets as goals to make. If several goals
> are specified, 'make' processes each of them in turn, in the order you name
> them. https://www.gnu.org/software/make/manual/html_node/Goals.html
>
> Ninja does not currently work for our project.
>
>
> It would be interesting to fix Ninja for your project then. Do you know
> what isn’t currently working?
> Also, Ninja has some APIs to expose all the dependency graph, which you
> could probably query somehow to find out which target you need to rebuild.
>
> Another possibility, albeit a bit different, could be to use a compilation
> cache to just rebuild everything but much faster, to the point that it
> might not be relevant anymore.
>
> /Florent
>
>
> On May 19, 2017, at 2:32 AM, Simon Richter 
> wrote:
>
> Hi,
>
> On 18.05.2017 23:48, Robert Patterson via cmake-developers wrote:
>
> My company has a large, predominately C++ codebase, with hundreds of
> targets, both for product and unit tests.  In an effort to improve the
> compile and test time for developers, which utilizes a continuous
> integration infrastructure, it is desirable to compile only the targets
> that are affected by developer's change sets.
>
>
> Erm, it should already work this way. If Make rebuilds a target that it
> shouldn't, the first step would be investigating why it thinks the
> target needs to be rebuilt.
>
> Dependency tracking is one of the oldest problems, and cmake should use
> an appropriate solution for the actual build system you use. The
> approach used for Make is a bit more conservative than you would require
> for GNU Make, but should nevertheless still work.
>
> http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
>
> is a good primer on dependency list generation with Make -- CMake
> generates rules that are similar to these.
>
> 'make' has a limitation where if 'make target1 target2 target3' is
> invoked, target1, target2, and target3 are built serially, not in
> parallel.
>
>
> Which version of make are you using?
>
>   Simon
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/

Re: [cmake-developers] Compile targets affected by changeset

[Florent Castelli]

> On 22 May 2017, at 20:07, Robert Patterson via cmake-developers 
>  wrote:
> 
> We understand that CMake and make already can rebuild targets which depend on 
> changed files, and this behavior works exactly as expected for us. Our issue 
> is not that make is rebuilding targets that it shouldn't. We would like this 
> 'compile targets which changed' behavior to work from a clean state, not just 
> for subsequent rebuilds. That is, from the first time cmake / make is 
> invoked, only the targets that depend on the given set of files will be 
> built, and no more.
> 
> When a developer submits a changeset to our continuous integration tool, a 
> fresh copy of the repository is checked out by the worker at the commit of 
> the changeset. We can access the set of changed files from the commit data. 
> Currently, cmake and make correctly identify that the system is in a clean 
> state and that everything should be built if the current top-level target is 
> specified. This is the point where we would like our behavior to differ. We 
> want the rebuild changed targets behavior as if everything were already 
> built, but from this clean state.
> 
> Once we have determined the targets to build, we could simply specify these 
> targets as goals to make in the command line, but we are hampered by the 
> following limitation of make.
> 
> We are using GNU make 3.82. Invoking 'make -j target1 target2 target3' on the 
> command line specifies the targets as goals to make. If several goals are 
> specified, 'make' processes each of them in turn, in the order you name them. 
> https://www.gnu.org/software/make/manual/html_node/Goals.html 
> 
> 
> Ninja does not currently work for our project.

It would be interesting to fix Ninja for your project then. Do you know what 
isn’t currently working?
Also, Ninja has some APIs to expose all the dependency graph, which you could 
probably query somehow to find out which target you need to rebuild.

Another possibility, albeit a bit different, could be to use a compilation 
cache to just rebuild everything but much faster, to the point that it might 
not be relevant anymore.

/Florent

> 
>> On May 19, 2017, at 2:32 AM, Simon Richter > > wrote:
>> 
>> Hi,
>> 
>> On 18.05.2017 23:48, Robert Patterson via cmake-developers wrote:
>> 
>>> My company has a large, predominately C++ codebase, with hundreds of
>>> targets, both for product and unit tests.  In an effort to improve the
>>> compile and test time for developers, which utilizes a continuous
>>> integration infrastructure, it is desirable to compile only the targets
>>> that are affected by developer's change sets.
>> 
>> Erm, it should already work this way. If Make rebuilds a target that it
>> shouldn't, the first step would be investigating why it thinks the
>> target needs to be rebuilt.
>> 
>> Dependency tracking is one of the oldest problems, and cmake should use
>> an appropriate solution for the actual build system you use. The
>> approach used for Make is a bit more conservative than you would require
>> for GNU Make, but should nevertheless still work.
>> 
>> http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/ 
>> 
>> 
>> is a good primer on dependency list generation with Make -- CMake
>> generates rules that are similar to these.
>> 
>>> 'make' has a limitation where if 'make target1 target2 target3' is
>>> invoked, target1, target2, and target3 are built serially, not in
>>> parallel.
>> 
>> Which version of make are you using?
>> 
>>   Simon
>> 
>> -- 
>> 
>> Powered by www.kitware.com
>> 
>> Please keep messages on-topic and check the CMake FAQ at: 
>> http://www.cmake.org/Wiki/CMake_FAQ
>> 
>> Kitware offers various services to support the CMake community. For more 
>> information on each offering, please visit:
>> 
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>> 
>> Visit other Kitware open-source projects at 
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake-developers
> 
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:

Re: [cmake-developers] Compile targets affected by changeset

[Brad King]

On 05/18/2017 05:48 PM, Robert Patterson via cmake-developers wrote:
>  we must have a preprocessing step (makedepend or gcc -M) 

Where in your example(s) does this step take place?

> 'make' has a limitation where if 'make target1 target2 target3' is invoked,
> target1, target2, and target3 are built serially, not in parallel.

I don't think that is a limitation of `make`, but rather of CMake's
generated makefiles.  The `Makefile` files in the build tree work
internally in such a way that only one target can be built from the
command-line at a time, so they contain a `.NOTPARALLEL` mark to tell
GNU make to act serially.

> To get around this, we decided that the best option was to dynamically
> create a new [cmake] target which depends on the affected targets.

Another approach is to use `CMakeFiles/Makefile2` directly:

  make cmake_check_build_system
  make -j -f CMakeFiles/Makefile2 dir1/tgt1.dir/all dir2/tgt2.dir/all 
dir3/tgt3.dir/all

This will build all of the targets in parallel safely.  The difficulty
will be in constructing the paths to these per-target `/all` make targets.
Maybe some changes to the Unix Makefiles generator could relieve that.

Or, we could just expose an alternative entry point in the Makefile files.
It is likely not too hard to get the following to work:

  make -j cmake_targets_variable TARGETS="tgt1 tgt2 tgt3"

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Compile targets affected by changeset

[Robert Patterson via cmake-developers]

We understand that CMake and make already can rebuild targets which depend on 
changed files, and this behavior works exactly as expected for us. Our issue is 
not that make is rebuilding targets that it shouldn't. We would like this 
'compile targets which changed' behavior to work from a clean state, not just 
for subsequent rebuilds. That is, from the first time cmake / make is invoked, 
only the targets that depend on the given set of files will be built, and no 
more.

When a developer submits a changeset to our continuous integration tool, a 
fresh copy of the repository is checked out by the worker at the commit of the 
changeset. We can access the set of changed files from the commit data. 
Currently, cmake and make correctly identify that the system is in a clean 
state and that everything should be built if the current top-level target is 
specified. This is the point where we would like our behavior to differ. We 
want the rebuild changed targets behavior as if everything were already built, 
but from this clean state.

Once we have determined the targets to build, we could simply specify these 
targets as goals to make in the command line, but we are hampered by the 
following limitation of make.

We are using GNU make 3.82. Invoking 'make -j target1 target2 target3' on the 
command line specifies the targets as goals to make. If several goals are 
specified, 'make' processes each of them in turn, in the order you name them. 
https://www.gnu.org/software/make/manual/html_node/Goals.html 


Ninja does not currently work for our project.

> On May 19, 2017, at 2:32 AM, Simon Richter  wrote:
> 
> Hi,
> 
> On 18.05.2017 23:48, Robert Patterson via cmake-developers wrote:
> 
>> My company has a large, predominately C++ codebase, with hundreds of
>> targets, both for product and unit tests.  In an effort to improve the
>> compile and test time for developers, which utilizes a continuous
>> integration infrastructure, it is desirable to compile only the targets
>> that are affected by developer's change sets.
> 
> Erm, it should already work this way. If Make rebuilds a target that it
> shouldn't, the first step would be investigating why it thinks the
> target needs to be rebuilt.
> 
> Dependency tracking is one of the oldest problems, and cmake should use
> an appropriate solution for the actual build system you use. The
> approach used for Make is a bit more conservative than you would require
> for GNU Make, but should nevertheless still work.
> 
> http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
> 
> is a good primer on dependency list generation with Make -- CMake
> generates rules that are similar to these.
> 
>> 'make' has a limitation where if 'make target1 target2 target3' is
>> invoked, target1, target2, and target3 are built serially, not in
>> parallel.
> 
> Which version of make are you using?
> 
>   Simon
> 
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more 
> information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Compile targets affected by changeset

[Ben Boeckel]

On Thu, May 18, 2017 at 17:48:26 -0400, Robert Patterson via cmake-developers 
wrote:
> 'make' has a limitation where if 'make target1 target2 target3' is
> invoked, target1, target2, and target3 are built serially, not in
> parallel.

Well, this makes sense since there's no `-j` flag given. Are you passing
a `-j` flag to make too?

>   To get around this, we decided that the best option was to
> dynamically create a new [cmake] target which depends on the affected
> targets. This is where we could use some guidance as we have two
> possible approaches: 1. This could either be implemented in cmake
> code; by supplying a command line argument, specifying the "dynamic"
> target which in turn would create a target during the generation
> process. 2. Explicitly specify the "dynamic" target in our project's
> CMakeLists.txt file, and allow cmake to simply add the affected
> targets as a dependency to the dynamic target.

Have you considered using the Ninja generator? It does have dependency
information right from the compiler (via the `-M` flag family).

--Ben
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] Compile targets affected by changeset

[Simon Richter]

Hi,

On 18.05.2017 23:48, Robert Patterson via cmake-developers wrote:

> My company has a large, predominately C++ codebase, with hundreds of
> targets, both for product and unit tests.  In an effort to improve the
> compile and test time for developers, which utilizes a continuous
> integration infrastructure, it is desirable to compile only the targets
> that are affected by developer's change sets.

Erm, it should already work this way. If Make rebuilds a target that it
shouldn't, the first step would be investigating why it thinks the
target needs to be rebuilt.

Dependency tracking is one of the oldest problems, and cmake should use
an appropriate solution for the actual build system you use. The
approach used for Make is a bit more conservative than you would require
for GNU Make, but should nevertheless still work.

http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

is a good primer on dependency list generation with Make -- CMake
generates rules that are similar to these.

> 'make' has a limitation where if 'make target1 target2 target3' is
> invoked, target1, target2, and target3 are built serially, not in
> parallel.

Which version of make are you using?

   Simon



signature.asc
Description: OpenPGP digital signature
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[cmake-developers] Compile targets affected by changeset

[Robert Patterson via cmake-developers]

Greetings,My company has a large, predominately C++ codebase, with hundreds of targets, both for product and unit tests.  In an effort to improve the compile and test time for developers, which utilizes a continuous integration infrastructure, it is desirable to compile only the targets that are affected by developer's change sets.Our approach leverages the cmake target source file information, along with the dependency graph, to identify which targets are directly affected by the changeset, and then trace the dependency graph to find all targets which depend on these files. We use 'make', so cmake does not analyze header information, so we must have a preprocessing step (makedepend or gcc -M) to identify which source files may be affected by the header files in the change set. These indirectly changed sources are fed back into the file changeset.'make' has a limitation where if 'make target1 target2 target3' is invoked, target1, target2, and target3 are built serially, not in parallel. To get around this, we decided that the best option was to dynamically create a new [cmake] target which depends on the affected targets. This is where we could use some guidance as we have two possible approaches: 1. This could either be implemented in cmake code; by supplying a command line argument, specifying the "dynamic" target which in turn would create a target during the generation process. 2. Explicitly specify the "dynamic" target in our project's CMakeLists.txt file, and allow cmake to simply add the affected targets as a dependency to the dynamic target.Our current (attached) patch adds an option (--change-set=) for to cmake to read a (newline separated with absolute paths) file which contains the list of files in the changeset. cmake parses the file and outputs the set of affected targets and then exits before makefile generation.We would like your opinions on how to best implement this. Would it be better to handle the creation of the dummy target in CMakeLists.txt, or to handle it in cmake code?  Additionally, would you be interested in adopting these changes upstream?option 1:cmake --change-set=mychanges.txt >> targets.txtcreate dummy target in CMakeLists.txtcmake (parses targets.txt and adds dependencies to dummy-target)make dummy-target pros: simple, small addition to cmakecons: less general, complexity moved to build scriptsoption 2:cmake --change-set=mychanges.txt >> targets.txtcmake --dependencies=targets.txt --dynamic-target=dummy-targetmake dummy-targetoption 3:cmake --change-set=mychanges.txt --dynamic-target=dummy-targetmake dummy-targetpros: more generalcons: more complexity introduced to cmake codeI appreciate your time and look forward to your feedback.--Robert Patterson

compile-changeset.patch
Description: Binary data
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers