Re: [cmake-developers] CMake daemon for user tools

[Stephen Kelly]

On 03/25/2016 09:25 AM, Tobias Hunger wrote:
> I am personally rather unsure about how to proceed. I can help make
> this branch production/merge ready, but I do not want to maintain it
> indefinitely after it is merged. It touches to many CMake internals
> for me to be comfortable hacking on that code. 

Yes, this lack of interest in general in maintaining it upstream is
currently a demotivator. I think we need a group of people interested in
doing that.

Brad/Ben, no response since I posted the blog or the code. Any word of
interest in any of it?

Thanks,

Steve.

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Vladimir Prus]

On 26-Jan-16 12:14 AM, Stephen Kelly wrote:

Vladimir Prus wrote:



http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/12658/focus=13004


Hi Stephen,

this sounds very much like what I has proposed for Boost.Build, see:

http://vladimirprus.com/talks/boosting-cdt-builds/


Yes, I'm familiar with that. It is listed as prior art in the gmane link you
quoted :)


:-)


I am scheduled to give a talk at FOSDEM about the feature and how user
tools can interact with it:



https://fosdem.org/2016/schedule/event/enabling_gui_tools_for_cmake_code

...

Is anyone interested enough in this potential cmake feature to
join the development effort?


You can guess I'm not very interested in cmake, but maybe we can discuss
the protocol, so that an IDE can work with different build systems? Think
we can have a chat at FOSDEM?


Yep, sounds good to me. I don't have a schedule decided yet for which talks
I'll attend, but I should be reachable by email to arrange to meet. I assume
you've seen

  https://steveire.wordpress.com/2016/01/24/cmake-daemon-for-user-tools/


Yes, that is quite impressive. I don't have a schedule yet either, but I am
going to attend your talk, so we can sync there.

--
Vladimir Prus
http://vladimirprus.com

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Vladimir Prus]

On 10-Jan-16 2:10 PM, Stephen Kelly wrote:


Hello,

I've been working on adding a daemon mode for cmake to provide
information to user tools - such as IDEs - about the buildsystem.

Following the discussion about providing metadata for IDEs to consume
I proposed creating a long-running process which would provide a protocol
to access information about the buildsystem, and about the content of the
cmake files themselves:

  
http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/12658/focus=13004


Hi Stephen,

this sounds very much like what I has proposed for Boost.Build, see:

http://vladimirprus.com/talks/boosting-cdt-builds/


I am scheduled to give a talk at FOSDEM about the feature and how user
tools can interact with it:

  https://fosdem.org/2016/schedule/event/enabling_gui_tools_for_cmake_code

...

Is anyone interested enough in this potential cmake feature to
join the development effort?


You can guess I'm not very interested in cmake, but maybe we can discuss the 
protocol,
so that an IDE can work with different build systems? Think we can
have a chat at FOSDEM?


--
Vladimir Prus
http://vladimirprus.com

--

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[James Johnston]

> -Original Message-
> From: Milian Wolff [mailto:m...@milianw.de]
> Sent: Saturday, January 23, 2016 15:41
> To: cmake-developers@cmake.org
> Cc: James Johnston
> Subject: Re: [cmake-developers] CMake daemon for user tools
> 
> You are aware that modern std::string is SSO'ed? I'm running on such a
> system.
> Another reason why you should not reinvent the wheel and keep relying on
> the STL wherever possible.
>

> Qt has such a class, it's called QVarLengthArray, and I've also been able
to
> apply it in multiple occasions to good effect. That said, when you look at

Yeah, but std::string is platform dependent, and the size of the buffer is
also platform dependent.  Maybe it tends to be optimal for CMake.  Then
again, maybe a larger buffer is needed.  I don't know.  The flexible option
would be something that does exactly like QVarLengthArray.  Different
variables might have different optimal sizes.

Some sample small strings for gcc/clang/VC++:
http://stackoverflow.com/a/28003328/562766
Note that none of them are large enough to store an absolute path, which are
maybe common (???) in CMake.  Also there's a fair bit of variation; if CMake
wants consistent performance in a section of code across compilers, it would
need a way to explicitly specify the small string size.  For example, some
are large enough to store typical target sizes - and some maybe are not.

There is also boost::container::small_vector in addition to QVarLengthArray:
http://www.boost.org/doc/libs/1_60_0/doc/html/boost/container/small_vector.h
tml

> Just run cmake (or the daemon) through a profiler and check the results.
> Doing so for the daemon (built with RelWithDebInfo) on the LLVM build dir
> and recording it with `perf --call-graph lbr` I get these hotspots when
looking
> at the results with `perf report -g graph --no-children`:
> 
> +8.67%  cmake  cmake[.]
> cmGlobalGenerator::FindGeneratorTargetImpl
> +4.21%  cmake  libc-2.22.so [.] _int_malloc
> +2.67%  cmake  cmake[.] cmCommandArgument_yylex
> +2.09%  cmake  libc-2.22.so [.] _int_free
> +2.06%  cmake  libc-2.22.so [.] __memcmp_sse4_1
> +1.84%  cmake  libc-2.22.so [.] malloc
> 
> This already shows you that you can gain a lot by reducing the number of
> allocations done. Heaptrack is a good tool for that.

Next question would be: who is calling malloc?  Or rather, what % of callers
are std::string, std::vector, other STL classes vs custom data structures?
Next question would be: what is the size of those mallocs, for each caller?

(Sorry I don't currently have an environment set up with a profiler to test
this myself at the moment.) 

> Similarly, someone should
> investigate cmGlobalGenerator::FindGeneratorTargetImpl. That does a lot of
> string comparisons to find targets from my quick glance, so indeed could
be
> sped up with a smarter string class.
> 
> But potentially you could also get a much quicker lookup by storing a hash
> map of target name to cmGeneratorTarget.

Indeed; there has got to be a way to reduce the complexity of that function
in number of targets compared, if not the low-level string comparison itself
as well.  For example, if target names are short-ish, the string class has
large enough SSO, and the underlying string class made use of vector CPU
instructions for comparison, there is probably very little to be gained
without such a hash map.  (On the other hand, if some of the previous
assumptions are not true on some common CMake platforms) 

> Seems like there's more than enough areas one could (and should) optimize
> CMake.

Indeed.  Another idea - probably unrelated to the string allocations issue,
but still - that came to mind: what if link-time code
generation/optimization is turned on?  IIRC this is not default in CMake.
Maybe CMake is sufficiently well-organized (e.g. small functions
implementations moved to header files) such that what needs to be inlined
across units, is already being inlined.  Then again, maybe it's not.  I've
seen other projects rely on this feature to keep clean organization by
keeping implementations in .CPP files without sacrificing performance, and
when you turn off LTCG performance takes a major hit... 

Also IIRC there are still a few optimizations that are turned off when CMake
is built with RelWithDebInfo instead of Release.  I forget the exact
specifics at the moment but e.g. on Visual C++ when you ask it to turn on
debug symbols, it will change the default values of some optimization flags.
So a cursory examination of the flags wouldn't reveal all cases.

However, one of my bigger performance gripes, being a primarily Windows
developer right now, is the process creation overhead, especially during
configuration.  I think that is completely dominating over any CMake code
being run 

Re: [cmake-developers] CMake daemon for user tools

[Milian Wolff]

On Freitag, 22. Januar 2016 17:55:46 CET James Johnston wrote:
> > -Original Message-
> > From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
> > On Behalf Of Milian Wolff
> > Sent: Thursday, January 21, 2016 22:31
> > To: Daniel Pfeifer
> > Cc: CMake Developers; Stephen Kelly
> > Subject: Re: [cmake-developers] CMake daemon for user tools
> > 
> > > What do you think about string interning? I started a cmString class
> > > [2] that stores short strings inplace and puts long strings into a
> > > pool. Copies never allocate extra memory and equality comparison is
> > > always O(1).
> > > In addition to `operator<` (which is lexicographical) there is a O(1)
> > > comparison that can be used where lexicographical ordering is not
> > > required (ie. for lookup tables).
> > > 
> > > [1] https://en.wikipedia.org/wiki/String_interning
> > > [2] https://github.com/purpleKarrot/CMake/commits/string-pool
> > 
> > Imo, you should replace this custom code by a boost::flyweight of
> 
> std::string.
> 
> > That said, I think this can have a significant impact on the memory
> 
> footprint
> 
> > of CMake, considering how heavy it relies on strings internally. But it
> 
> also
> 
> > seems to mutate strings a lot. I've seen places e.g. where a list of
> 
> compile-
> 
> > time known identifiers is prepended with "CMAKE_" at runtime. This is slow
> > with normal strings (reallocations), but will also be slow with a
> 
> flyweight or
> 
> > interning, possibly even leading to the pollution of the internal pool
> 
> with
> 
> > temporary strings.
> > 
> > Did you measure any impact on both, runtime speed and memory footprint
> > yet?
> 
> I was wondering the same.  I would guess maybe the biggest impact would be
> the inplace storage of strings for small sized strings.  But to know the
> inplace buffer size would probably require some profiling and measurement of
> string sizes... otherwise it is just a wild guess...

You are aware that modern std::string is SSO'ed? I'm running on such a system. 
Another reason why you should not reinvent the wheel and keep relying on the 
STL wherever possible.

> Maybe for testing, you can swap out the string header file on your system
> with one that logs allocations/string sizes, and perhaps also profiles the
> time it takes to make each allocation?

The data recorded by heaptrack would allow such an analysis post-mortem 
without modification of any header. Someone just needs to write such an 
analysis step...

> The interesting question is: could inplace storage be used for 95% of the
> cases such that fussing with string interning becomes unnecessary
> complexity?  If so, then you mentioned equality comparison as another issue:
> the interesting question there is how much time is spent on allocations vs
> comparisons...

Just run cmake (or the daemon) through a profiler and check the results. Doing 
so for the daemon (built with RelWithDebInfo) on the LLVM build dir and 
recording it with `perf --call-graph lbr` I get these hotspots when looking at 
the results with `perf report -g graph --no-children`:

+8.67%  cmake  cmake[.] 
cmGlobalGenerator::FindGeneratorTargetImpl
+4.21%  cmake  libc-2.22.so [.] _int_malloc 

   
+2.67%  cmake  cmake[.] cmCommandArgument_yylex
+2.09%  cmake  libc-2.22.so [.] _int_free   

   
+2.06%  cmake  libc-2.22.so [.] __memcmp_sse4_1
+1.84%  cmake  libc-2.22.so [.] malloc

This already shows you that you can gain a lot by reducing the number of 
allocations done. Heaptrack is a good tool for that. Similarly, someone should 
investigate cmGlobalGenerator::FindGeneratorTargetImpl. That does a lot of 
string comparisons to find targets from my quick glance, so indeed could be 
sped up with a smarter string class.

But potentially you could also get a much quicker lookup by storing a hash map 
of target name to cmGeneratorTarget.

> In another application I worked on, I was able to get a big improvement in
> performance by replacing usage of std::vector in one place with a custom
> vector that stack-allocated the first 10 items (i.e. fixed-size C array as a
> member variable of the class), and then reverted to a regular vector after
> that.  But to pick the number "10" required some profiling/measure

Re: [cmake-developers] CMake daemon for user tools

[James Johnston]

> -Original Message-
> From: cmake-developers [mailto:cmake-developers-boun...@cmake.org]
> On Behalf Of Milian Wolff
> Sent: Thursday, January 21, 2016 22:31
> To: Daniel Pfeifer
> Cc: CMake Developers; Stephen Kelly
> Subject: Re: [cmake-developers] CMake daemon for user tools
> 
> > What do you think about string interning? I started a cmString class
> > [2] that stores short strings inplace and puts long strings into a
> > pool. Copies never allocate extra memory and equality comparison is
> > always O(1).
> > In addition to `operator<` (which is lexicographical) there is a O(1)
> > comparison that can be used where lexicographical ordering is not
> > required (ie. for lookup tables).
> >
> > [1] https://en.wikipedia.org/wiki/String_interning
> > [2] https://github.com/purpleKarrot/CMake/commits/string-pool
> 
> Imo, you should replace this custom code by a boost::flyweight of
std::string.
> That said, I think this can have a significant impact on the memory
footprint
> of CMake, considering how heavy it relies on strings internally. But it
also
> seems to mutate strings a lot. I've seen places e.g. where a list of
compile-
> time known identifiers is prepended with "CMAKE_" at runtime. This is slow
> with normal strings (reallocations), but will also be slow with a
flyweight or
> interning, possibly even leading to the pollution of the internal pool
with
> temporary strings.
> 
> Did you measure any impact on both, runtime speed and memory footprint
> yet?

I was wondering the same.  I would guess maybe the biggest impact would be
the inplace storage of strings for small sized strings.  But to know the
inplace buffer size would probably require some profiling and measurement of
string sizes... otherwise it is just a wild guess... 

Maybe for testing, you can swap out the string header file on your system
with one that logs allocations/string sizes, and perhaps also profiles the
time it takes to make each allocation?

The interesting question is: could inplace storage be used for 95% of the
cases such that fussing with string interning becomes unnecessary
complexity?  If so, then you mentioned equality comparison as another issue:
the interesting question there is how much time is spent on allocations vs
comparisons...

In another application I worked on, I was able to get a big improvement in
performance by replacing usage of std::vector in one place with a custom
vector that stack-allocated the first 10 items (i.e. fixed-size C array as a
member variable of the class), and then reverted to a regular vector after
that.  But to pick the number "10" required some profiling/measurement.  The
remaining use of the heap was so negligible as to not be worth improving.

Best regards,

James Johnston


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Brad King]

On 01/20/2016 04:03 PM, Stephen Kelly wrote:
> FYI I merged a reduce-allocations branch to next for testing. It came from 
> Milian here:
>  
>  https://github.com/steveire/CMake/pull/1

Very nice!

Thanks,
-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Daniel Pfeifer]

On Wed, Jan 20, 2016 at 10:03 PM, Stephen Kelly  wrote:
> Milian Wolff wrote:
>
>>> I'm concerned that the memory usage of a daemon implementing the proposed
>>> capabilities may be too large to be practical (at least without a major
>>> redesign of certain structures that tend to duplicate substrings, or
>>> some kind of out-of-core approach).
>>
>> This sounds like optimizations to that daemon will benefit CMake.
>
> FYI I merged a reduce-allocations branch to next for testing. It came from
> Milian here:
>
>  https://github.com/steveire/CMake/pull/1

Very nice! It is impressive what kind of an impact a simple 'const&'
can have. Also, nice work identifying the hotspots.

> Milian also mentioned the possibility of using something like a sqlite
> database (probably what you meant by out-of-core above) for definitions for
> querying by the daemon.
>
> I also mentioned some possibile optimization possibilities, such as removing
> the closure of definitions created for all directories. I wrote a branch
> which does that some months ago, but it resulted in a slow down. I'll see if
> I can rebase the commit to master and push it to github, together with a
> patch for avoiding computing the hash multiple times in cmDefinitions.

What do you think about string interning? I started a cmString class
[2] that stores short strings inplace and puts long strings into a
pool. Copies never allocate extra memory and equality comparison is
always O(1).
In addition to `operator<` (which is lexicographical) there is a O(1)
comparison that can be used where lexicographical ordering is not
required (ie. for lookup tables).

[1] https://en.wikipedia.org/wiki/String_interning
[2] https://github.com/purpleKarrot/CMake/commits/string-pool

cheers, Daniel
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Stephen Kelly]

Milian Wolff wrote:

>> I'm concerned that the memory usage of a daemon implementing the proposed
>> capabilities may be too large to be practical (at least without a major
>> redesign of certain structures that tend to duplicate substrings, or
>> some kind of out-of-core approach).
> 
> This sounds like optimizations to that daemon will benefit CMake.

FYI I merged a reduce-allocations branch to next for testing. It came from 
Milian here:
 
 https://github.com/steveire/CMake/pull/1

Milian also mentioned the possibility of using something like a sqlite 
database (probably what you meant by out-of-core above) for definitions for 
querying by the daemon. 

I also mentioned some possibile optimization possibilities, such as removing 
the closure of definitions created for all directories. I wrote a branch 
which does that some months ago, but it resulted in a slow down. I'll see if 
I can rebase the commit to master and push it to github, together with a 
patch for avoiding computing the hash multiple times in cmDefinitions. 

Thanks,

Steve.


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Stephen Kelly]

Tobias Hunger wrote:

> So could we please get a *documented* way that an IDE can *rely on to
> be available* that provides basic information on a
> project/configuration:
> 
> * Source directory, build directory
> * Files that belong to the project (sources, headers, custom cmake
> modules, CMakeLists.txts, resources, ...)
> * Files that belong to the build system (anything that needs watching
> to re-generate the IDE-integration information at the right times)
> * Targets that can be built
> * Files that belong to each target
> * Compiler flags
> * Defines in effect
> * Include paths used
> * The compilers used to build those files

Yes, that's the kind of thing my branch on github provides. That's why I 
worked on those things first.

Did you look at my github branch already by the way?

Thanks,

Steve.


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Alexander Neundorf]

On Tuesday, January 12, 2016 23:20:14 Milian Wolff wrote:
> On Montag, 11. Januar 2016 23:22:23 CET Alexander Neundorf wrote:
...
> > Stephens big approach will need some time until it is ready, while such a
> > (relatively) simple thing can probably be done within one release cycle
> > (but I don't have the time to do it).
> > 
> > I don't know what the others here think about adding your json-approach or
> > not.
> 
> I don't think I'll have the required time to help here either, but I agree
> with your reasoning: Your patch is the potentially quickest to get something
> done. But it only fixes the usability issue of generators, it does not come
> with a useable generator which we'll still need to write.

I thought Aleix's patch was already quite good ?

Alex

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Alexander Neundorf]

On Monday, January 11, 2016 15:59:35 Aleix Pol wrote:
...
> 
> Hi Stephen, everyone,
> I've already discussed this in private with you. I think it's a good
> idea and I'd like to make sure we can benefit from this.
> 
> I'm unsure of the feasibility of the project though. 

you maybe remember that my main issue with your patch last year was that it's 
not a "generator", while it does what generators are for.
After talking with Milian a bit several week ago, I understood that an 
important reason why the kdevelop team does not want a generator is that there 
is only one generator per build-tree, i.e. the user has to decide beforehand 
which IDE project files he wants.
With the branch I created on github a few days ago, which I announced here 
too, I changed the way the extra generators are activated: they are not 
activated like the main generators anymore, but can be turned on and off using 
normal cmake cache options, so multiple generators (e.g. the generic json 
generator and let's say the Eclipse generator) can both be turned on, 
initially or later on. I think this should solve this problem.
The work there 
(https://github.com/neundorf/CMake/tree/RefactorExtraGeneratorsExperimental ) 
is not ready for inclusing, but it shows how it can be done and can serve as a 
starting point/inspiration.

Stephens big approach will need some time until it is ready, while such a 
(relatively) simple thing can probably be done within one release cycle (but I 
don't have the time to do it).

I don't know what the others here think about adding your json-approach or 
not.

Alex

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] CMake daemon for user tools

[Aleix Pol]

On Sun, Jan 10, 2016 at 12:10 PM, Stephen Kelly  wrote:
>
> Hello,
>
> I've been working on adding a daemon mode for cmake to provide
> information to user tools - such as IDEs - about the buildsystem.
>
> Following the discussion about providing metadata for IDEs to consume
> I proposed creating a long-running process which would provide a protocol
> to access information about the buildsystem, and about the content of the
> cmake files themselves:
>
>  
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/12658/focus=13004
>
> The original post in that thread lists design goals.
>
> This design is independent of any generator, and compared to the solution of
> writing json to a file, this solution doesn't require generating
> anything that is
> not asked for.  This also side-steps the problem of 'stale' files remaining
> in the build directory, and the problem of needing to trigger re-generation
> of the file, or watching it for changes.  This solution also allows
> parameters (eg the config) to be specified in requests, side-stepping a
> difficulty we encountered before to represent things like that.  This
> solution also does not require actually generating the buildsystem
> files. Only the configure and compute steps of cmake are executed.
>
> I am scheduled to give a talk at FOSDEM about the feature and how user
> tools can interact with it:
>
>  https://fosdem.org/2016/schedule/event/enabling_gui_tools_for_cmake_code
>
> with a preview in Berlin:
>
>  http://www.meetup.com/berlincplusplus/events/227896427/
>
> I have now also pushed a branch to my github clone with the start of the
> server mode:
>
>  https://github.com/steveire/CMake/commits/cmake-daemon
>
> Currently the branch only makes targets and backtraces etc available via
> the protocol.  I have also created a plan for extending the protocol to
> make code completion and variable debugging possible, and analysed the
> depedencies of the tasks:
>
>  http://www.steveire.com/cmake-daemon-tasks.png
>
> However, I can't complete those tasks myself: I don't have
> relevant experience building IDEs to know how best to design the
> protocol, what IDE tools really need, how to design a fail-safe parser
> etc.  Additionally, I think a design which incorporates design ideas
> and implementation from more than one person will be better in the end.
>
> So, this needs to be a collaborative effort if it is to go anywhere, with
> more people writing commits in the cmake repo.
>
> The above (copied below) task list would be enough to get a read-only
> browser of a cmake project quite quickly.
>
> Following that, an effort would be needed to handle the difference
> between the dirty state of an editor and the on-disk file content. That's
> a solvable problem and together with filesystem notifications provided
> by libuv, it will lead to features like code completion.
>
> Also necessary eventually would be to make the cmake parser more
> fail-safe, so that information is still available if the user
> had not yet closed a command argument list for example, and make the
> processing more fail-safe in the case where they have written
> add_subdirectory(foo), but have not yet created foo/CMakeLists.txt.
>
> I would be looking to people who work on IDEs for guidance on how to do
> things like that, as some of them are likely already for handling C++
> code in the IDEs.
>
> So, this needs to be a collaborative effort. I can provide guidance to
> anyone who wishes to participate in changing the source code of CMake
> to make it possible.
>
> Is anyone interested enough in this potential cmake feature to
> join the development effort?
>
> Thanks,
>
> Steve.
>
>
> * Provide basic daemon mode with event loop and communication protocol
> -- Written, but not reviewed
> * Provide protocol to query the buildsystem structure
> -- Written, but not complete or reviewed
> * Extend cmListFileParser to record location of start and end of commands
> * Rewrite cmIfCommand to record which branches are executed
> * Store the location of start and end of user-defined functions
> * Provide protocol to query for unexecuted lines
> * Define a new snapshot type to occur at an arbitrary point
> * Create snapshots after branching to other files (after an include() or
> find_package() or add_subdirectory etc)
> * Create snapshots after points with external effects (file(WRITE),
> configure_file(), execute_process etc)
> * Find most recent snapshot occuring before a specified line
> * Implement feature to start at a specific snapshot and execute commands
> up to a particular line
> * Make server find correct snapshot to start at and execute required
> commands up to specified line
> * Provide protocol to dump variable defintions at a particular line
> * Extract implementation of IssueMessage from cmMakefile and
> cmLocalGenerator into cmake
> * Extract computed target properties from cmTarget::GetProperty into
> cmComputedTargetProperties
> * Implement 

[cmake-developers] CMake daemon for user tools

[Stephen Kelly]

Hello,

I've been working on adding a daemon mode for cmake to provide
information to user tools - such as IDEs - about the buildsystem.

Following the discussion about providing metadata for IDEs to consume
I proposed creating a long-running process which would provide a protocol
to access information about the buildsystem, and about the content of the
cmake files themselves:

 
http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/12658/focus=13004

The original post in that thread lists design goals.

This design is independent of any generator, and compared to the solution of
writing json to a file, this solution doesn't require generating
anything that is
not asked for.  This also side-steps the problem of 'stale' files remaining
in the build directory, and the problem of needing to trigger re-generation
of the file, or watching it for changes.  This solution also allows
parameters (eg the config) to be specified in requests, side-stepping a
difficulty we encountered before to represent things like that.  This
solution also does not require actually generating the buildsystem
files. Only the configure and compute steps of cmake are executed.

I am scheduled to give a talk at FOSDEM about the feature and how user
tools can interact with it:

 https://fosdem.org/2016/schedule/event/enabling_gui_tools_for_cmake_code

with a preview in Berlin:

 http://www.meetup.com/berlincplusplus/events/227896427/

I have now also pushed a branch to my github clone with the start of the
server mode:

 https://github.com/steveire/CMake/commits/cmake-daemon

Currently the branch only makes targets and backtraces etc available via
the protocol.  I have also created a plan for extending the protocol to
make code completion and variable debugging possible, and analysed the
depedencies of the tasks:

 http://www.steveire.com/cmake-daemon-tasks.png

However, I can't complete those tasks myself: I don't have
relevant experience building IDEs to know how best to design the
protocol, what IDE tools really need, how to design a fail-safe parser
etc.  Additionally, I think a design which incorporates design ideas
and implementation from more than one person will be better in the end.

So, this needs to be a collaborative effort if it is to go anywhere, with
more people writing commits in the cmake repo.

The above (copied below) task list would be enough to get a read-only
browser of a cmake project quite quickly.

Following that, an effort would be needed to handle the difference
between the dirty state of an editor and the on-disk file content. That's
a solvable problem and together with filesystem notifications provided
by libuv, it will lead to features like code completion.

Also necessary eventually would be to make the cmake parser more
fail-safe, so that information is still available if the user
had not yet closed a command argument list for example, and make the
processing more fail-safe in the case where they have written
add_subdirectory(foo), but have not yet created foo/CMakeLists.txt.

I would be looking to people who work on IDEs for guidance on how to do
things like that, as some of them are likely already for handling C++
code in the IDEs.

So, this needs to be a collaborative effort. I can provide guidance to
anyone who wishes to participate in changing the source code of CMake
to make it possible.

Is anyone interested enough in this potential cmake feature to
join the development effort?

Thanks,

Steve.


* Provide basic daemon mode with event loop and communication protocol
-- Written, but not reviewed
* Provide protocol to query the buildsystem structure
-- Written, but not complete or reviewed
* Extend cmListFileParser to record location of start and end of commands
* Rewrite cmIfCommand to record which branches are executed
* Store the location of start and end of user-defined functions
* Provide protocol to query for unexecuted lines
* Define a new snapshot type to occur at an arbitrary point
* Create snapshots after branching to other files (after an include() or
find_package() or add_subdirectory etc)
* Create snapshots after points with external effects (file(WRITE),
configure_file(), execute_process etc)
* Find most recent snapshot occuring before a specified line
* Implement feature to start at a specific snapshot and execute commands
up to a particular line
* Make server find correct snapshot to start at and execute required
commands up to specified line
* Provide protocol to dump variable defintions at a particular line
* Extract implementation of IssueMessage from cmMakefile and
cmLocalGenerator into cmake
* Extract computed target properties from cmTarget::GetProperty into
cmComputedTargetProperties
* Implement cmGeneratorTarget::GetProperties in terms of
cmComputedTargetProperties
* Move implementation of cmTarget::{Get,Set,Append}Property to cmState
* Extract all IssueMessage parts of cmake into a cmMessenger
* Extract the cmSystemTools::Message part of cmMessenger into a virtual