Re: [cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

> > 2) Use libuv instead

I just saw the libuv library in the CMake sources.
 
> libuv for process management is on the list. I think it is waiting for
> porting to all of CMake's platforms to actually happen.

Does that mean it's only there for some specific scenarios and shouldn't
be used in general?

> I'm seeing this PR hung up on process:
> 
> https://github.com/libuv/libuv/pull/1527
> 
> I don't know how much followup work there is. I thought there was a
> CMake issue tracking it, but I don't see one.

IIUC this is about the CPU affinity for child processes?
Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child
processes are running and to get a notification when one has finished.
I think that this level of control is available in libuv already.

Sebastian

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Ben Boeckel]

On Fri, Jan 05, 2018 at 20:34:53 +0100, Sebastian Holtermann wrote:
> 2) Use libuv instead

libuv for process management is on the list. I think it is waiting for
porting to all of CMake's platforms to actually happen. I'm seeing this
PR hung up on process:

https://github.com/libuv/libuv/pull/1527

I don't know how much followup work there is. I thought there was a
CMake issue tracking it, but I don't see one.

--Ben
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[clinton]

Cool.  I'd like to have parallel automoc and uic.

How about this 4th option?
Don't use threads.  A single thread is able to spawn multiple processes, and 
wait on multiple processes, and react when 1 or more processes change state.  
I'm probably not familiar enough with kwsysProcess to know if this approach 
will work, but it might be easier and simpler(?) than making it thread safe.

Clint

- On Jan 5, 2018, at 12:34 PM, Sebastian Holtermann sebl...@xwmw.org wrote:

> Hello!
> 
> As you might have noticed I tried to parallelize AUTOMOC/UIC.
> https://gitlab.kitware.com/cmake/cmake/merge_requests/1632
> The issue that's blocking it now is that the kwsysProcess framework
> isn't thread safe. As a consequence it is not possible for threads to
> start processes concurrently using the kwsysProcess framework.
> 
> There are three options I see
> 
> 1) Make kwsysProcess thread safe
> 
> I don't think it's impossible but it's not a small task.
> Looking through the code I found a few issues that
> would block the whole idea for me.
>  - OpenVMS: I have little to no knowledge about OpenVMS.
>Wikipedia says it is basically abandoned.
>Can the OpenVMS code be removed? I couldn't test it anyway.
> 
>  - C vs. C++
>There's a lot of C in processUNIX.c (list allocation etc.).
>I would prefer to use C++11 (or higher), especially
>std::array, std::vector, std::thread,  std::mutex, etc..
>Is C++11 acceptable in kwsysProcess?
>This would make processUNIX.c processUNIX.cpp?
> 
> 2) Use libuv instead
> 
> Using the libuv event loop is overkill and would probably imply that
> more threads are started than necessary
> (the libuv thread pool will be started anyway).
> But it would allow to start multiple concurrent  (moc/uic) processes.
> Is there a reason to not use libuv?
> 
> 3) Abandon the idea to parallelize AUTOMOC/UIC
> 
> Well, I liked the idea.
> 
> 
> Any thoughts?
> 
> -Sebastian
> 
> --
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake-developers
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

On Freitag, 5. Januar 2018 13:00:30 CET clin...@elemtech.com wrote:
> Cool.  I'd like to have parallel automoc and uic.
> 
> How about this 4th option?
> Don't use threads.  A single thread is able to spawn multiple processes, and
> wait on multiple processes, and react when 1 or more processes change
> state.  I'm probably not familiar enough with kwsysProcess to know if this
> approach will work, but it might be easier and simpler(?) than making it
> thread safe.
> 
> Clint
> 

AFAIK there are two interfaces inside CMake that allow to start a process,
kwsysProcess and libuv.

kwsysProcess allows to start a single process and wait for it to finish. I 
thinks it's also possible to start multiple parallel processes.
But there's no interface to select/poll a number of parallel processes.

So the 4th option would boil down to use libuv for process management.

Sebastian

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

Hello!

As you might have noticed I tried to parallelize AUTOMOC/UIC.
https://gitlab.kitware.com/cmake/cmake/merge_requests/1632
The issue that's blocking it now is that the kwsysProcess framework
isn't thread safe. As a consequence it is not possible for threads to
start processes concurrently using the kwsysProcess framework.

There are three options I see

1) Make kwsysProcess thread safe

I don't think it's impossible but it's not a small task.
Looking through the code I found a few issues that
would block the whole idea for me.
  - OpenVMS: I have little to no knowledge about OpenVMS. 
Wikipedia says it is basically abandoned. 
Can the OpenVMS code be removed? I couldn't test it anyway.

  - C vs. C++
There's a lot of C in processUNIX.c (list allocation etc.).
I would prefer to use C++11 (or higher), especially
std::array, std::vector, std::thread,  std::mutex, etc..
Is C++11 acceptable in kwsysProcess?
This would make processUNIX.c processUNIX.cpp?

2) Use libuv instead

Using the libuv event loop is overkill and would probably imply that
more threads are started than necessary
(the libuv thread pool will be started anyway).
But it would allow to start multiple concurrent  (moc/uic) processes.
Is there a reason to not use libuv?

3) Abandon the idea to parallelize AUTOMOC/UIC

Well, I liked the idea.


Any thoughts?

-Sebastian
 
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS

[John Wordsworth]

I realise I wasn't very clear in my previous explanation. Hopefully this
paints a better picture;

- We have 40 or so developers spread across Visual Studio 2015 (MSBuild -
Windows), Xcode (macOS) and Make/Clang (Linux). 80% use Windows.
- We have a core set of libs in use by a number of projects (~30 static
libs and ~10 different projects), but the projects mix and match which libs
they use - merging the static libs (beyond a few of them) isn't something
we want to do.
- I have recently been revising our CMake structure and have boiled it down
to a fairly simple "add_library(x), target_link_libraries(x,
default_compile_settings, dependencies)" which means compile settings,
include directories and macros propagate through the libs nicely.
- We sometimes want to build and use certain libs as shared libs instead of
static libs.

With regards to the COMPILE_DEPENDS feature that I have mocked up and
started testing with our team;

- If and only if COMPILE_DEPENDS is set for a STATIC library, then when
building the target dependencies for that library, it uses the list of
provided targets instead of those that would have been inferred from
previous calls to "target_link_libraries" or "add_dependencies".

The reality is, the libs which are being built statically can nearly all
build in parallel with only one or two "real" dependencies. I understand
that our case is rather specific, but having implemented "COMPILE_DEPENDS"
and written ~10 lines of CMake in our project (hard coding the one or two
actual build order dependencies and disabling the rest), our compile graph
looks much nicer and saves a significant amount of time building on a
single machine (see https://www.johnwordsworth.com/temp/cmake_compile_
depends.jpg). We see similar savings on macOS using Xcode but have not
tested on Linux yet. When we allow Incredibuild to distribute building of
compilation units across 15-20 agents, the proportional build time drops
more dramatically between the two (the same project goes from ~2.6mins ->
~1.6mins).

I understand that just "hard overriding" the target dependencies is a bit
messy, but I'd also be happy to explore other ways I could add a similar
feature to CMake if there is a potentially better way to do this. What I am
ideally looking for in a solution is that I can continue to propagate
include directories / compile settings / preprocessor macros through the
chain of libs and that it improves building static libs in parallel across
all our build systems (MSVC / XCode / make). Ideally, I wouldn't have to
restructure my CMake project too much either - as the draft I have for our
new structure feels super clean now and refactoring it just to remove build
order dependencies would be shame. If there are any alternative ideas for
how to implement this, I'd love to discuss. I'm no CMake expert, but maybe
if we could use "target_link_libraries" with OBJECT libraries to grab
compile settings, or perhaps I could look into a mechanism for not adding
build order dependencies on VS2015/Xcode if there is no need too?

Thanks again for the feedback and discussion so far.

On Thu, Jan 4, 2018 at 12:16 PM, Eric Noulard 
wrote:

>
> 2018-01-04 10:48 GMT+01:00 Craig Scott :
>
>>
>>
>> On Thu, Jan 4, 2018 at 8:27 AM, John Wordsworth 
>> wrote:
>>
>>> I have recently been reviewing ways to improve build times for our
>>> project, which is comprised of a number of static libraries. I stumbled
>>> across this post on the CMake tracker from 2012/13 (
>>> https://cmake.org/Bug/view.php?id=13799). It suggests adding a
>>> COMPILE_DEPENDS target property to to explicitly set dependencies for
>>> STATIC libraries instead of always using all linked libraries as
>>> build-order dependencies.
>>>
>>> Having done a draft implementation in a local CMake repository it has
>>> shaved off  20% of our 120s build time. I expect the savings to be much
>>> more dramatic when I test with Incredibuild (approximately 50% based on
>>> tests done previously from just deleting dependencies manually in Visual
>>> Studio).
>>>
>>
> You said you tested with Incredibuild but with what kind of configuration?
> AFAIK Incredibuild takes its power from distributing the build? So does
> your CMake test uses something that distribute the build too ?
>
>
>>
>>> I don’t really want to refactor our code to use “OBJECT” libraries as
>>> the inability to link with other targets means that propagating compile
>>> options / include directories etc down the chain of linked libs becomes
>>> painful. This method allows me to switch between static and shared libs
>>> using a config option and none of my CMake scripts need to change.
>>>
>>
>> There's a couple more choices here. If your project consists of lots of
>> small (static) libraries, consider whether you can combine some of them to
>> result in a smaller number of larger libraries. This isn't always a gain,
>> but in terms of ability to