Re: [cmake-developers] kwsysProcess threadsafety

[Alexander Neundorf]

On 2018 M01 8, Mon 10:32:22 CET Jean-Michaël Celerier wrote:
> > 3) Abandon the idea to parallelize AUTOMOC/UIC
> 
> please no ! moc is a huge bottleneck in my builds (to the point that using
> verdigris  instead ends up having
> faster build time overall).

you could also just not use automoc and use the "manual" moc-wrapper macros. 
This should make the build-time overhead go away.
Instead the developer has to add the header file to be moc'ed to the 
CMakeLists.txt (personally I consider this acceptable).

Alex

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Brad King]

On 01/08/2018 07:26 AM, Sebastian Holtermann wrote:
> Then I'm going to refactor cmQtAutoGeneratorMocUic
> to use a libuv event loop internally.

Great.  Take a look at Source/cmUVHandlePtr.h for some C++ helpers.

-Brad
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

On Montag, 8. Januar 2018 06:56:57 CET Brad King wrote:
> On 01/06/2018 02:31 AM, Sebastian Holtermann wrote:
> >>> 2) Use libuv instead
> > 
> > I just saw the libuv library in the CMake sources.
> > 
> >> libuv for process management is on the list. I think it is waiting for
> >> porting to all of CMake's platforms to actually happen.
> > 
> > Does that mean it's only there for some specific scenarios and shouldn't
> > be used in general?
> 
> A goal is to eventually fully replace use of KWSys Process with libuv.
> I'd prefer not to make any major changes to KWSys Process in the meantime.
> 
> Currently libuv is used only for the server mode, but soon it will be
> used for CTest too:
> 
>   https://gitlab.kitware.com/cmake/cmake/merge_requests/1455
> 
> > Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child
> > processes are running and to get a notification when one has finished.
> 
> Use of libuv's process management for that would be good.  It supports
> fully asynchronous operation and so can manage many child processes from
> one thread.

Okay.
Then I'm going to refactor cmQtAutoGeneratorMocUic
to use a libuv event loop internally.

-Sebastian

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Brad King]

On 01/06/2018 02:31 AM, Sebastian Holtermann wrote:
>>> 2) Use libuv instead
> 
> I just saw the libuv library in the CMake sources.
>  
>> libuv for process management is on the list. I think it is waiting for
>> porting to all of CMake's platforms to actually happen.
> 
> Does that mean it's only there for some specific scenarios and shouldn't
> be used in general?

A goal is to eventually fully replace use of KWSys Process with libuv.
I'd prefer not to make any major changes to KWSys Process in the meantime.

Currently libuv is used only for the server mode, but soon it will be
used for CTest too:

  https://gitlab.kitware.com/cmake/cmake/merge_requests/1455

> Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child
> processes are running and to get a notification when one has finished.

Use of libuv's process management for that would be good.  It supports
fully asynchronous operation and so can manage many child processes from
one thread.

Thanks,
-Brad
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Jean-Michaël Celerier]

> 3) Abandon the idea to parallelize AUTOMOC/UIC

please no ! moc is a huge bottleneck in my builds (to the point that using
verdigris  instead ends up having
faster build time overall).

Best,
Jean-Michaël



---
Jean-Michaël Celerier
http://www.jcelerier.name

On Fri, Jan 5, 2018 at 8:34 PM, Sebastian Holtermann 
wrote:

> Hello!
>
> As you might have noticed I tried to parallelize AUTOMOC/UIC.
> https://gitlab.kitware.com/cmake/cmake/merge_requests/1632
> The issue that's blocking it now is that the kwsysProcess framework
> isn't thread safe. As a consequence it is not possible for threads to
> start processes concurrently using the kwsysProcess framework.
>
> There are three options I see
>
> 1) Make kwsysProcess thread safe
>
> I don't think it's impossible but it's not a small task.
> Looking through the code I found a few issues that
> would block the whole idea for me.
>   - OpenVMS: I have little to no knowledge about OpenVMS.
> Wikipedia says it is basically abandoned.
> Can the OpenVMS code be removed? I couldn't test it anyway.
>
>   - C vs. C++
> There's a lot of C in processUNIX.c (list allocation etc.).
> I would prefer to use C++11 (or higher), especially
> std::array, std::vector, std::thread,  std::mutex, etc..
> Is C++11 acceptable in kwsysProcess?
> This would make processUNIX.c processUNIX.cpp?
>
> 2) Use libuv instead
>
> Using the libuv event loop is overkill and would probably imply that
> more threads are started than necessary
> (the libuv thread pool will be started anyway).
> But it would allow to start multiple concurrent  (moc/uic) processes.
> Is there a reason to not use libuv?
>
> 3) Abandon the idea to parallelize AUTOMOC/UIC
>
> Well, I liked the idea.
>
>
> Any thoughts?
>
> -Sebastian
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake-developers
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

> > 2) Use libuv instead

I just saw the libuv library in the CMake sources.
 
> libuv for process management is on the list. I think it is waiting for
> porting to all of CMake's platforms to actually happen.

Does that mean it's only there for some specific scenarios and shouldn't
be used in general?

> I'm seeing this PR hung up on process:
> 
> https://github.com/libuv/libuv/pull/1527
> 
> I don't know how much followup work there is. I thought there was a
> CMake issue tracking it, but I don't see one.

IIUC this is about the CPU affinity for child processes?
Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child
processes are running and to get a notification when one has finished.
I think that this level of control is available in libuv already.

Sebastian

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Ben Boeckel]

On Fri, Jan 05, 2018 at 20:34:53 +0100, Sebastian Holtermann wrote:
> 2) Use libuv instead

libuv for process management is on the list. I think it is waiting for
porting to all of CMake's platforms to actually happen. I'm seeing this
PR hung up on process:

https://github.com/libuv/libuv/pull/1527

I don't know how much followup work there is. I thought there was a
CMake issue tracking it, but I don't see one.

--Ben
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[clinton]

Cool.  I'd like to have parallel automoc and uic.

How about this 4th option?
Don't use threads.  A single thread is able to spawn multiple processes, and 
wait on multiple processes, and react when 1 or more processes change state.  
I'm probably not familiar enough with kwsysProcess to know if this approach 
will work, but it might be easier and simpler(?) than making it thread safe.

Clint

- On Jan 5, 2018, at 12:34 PM, Sebastian Holtermann sebl...@xwmw.org wrote:

> Hello!
> 
> As you might have noticed I tried to parallelize AUTOMOC/UIC.
> https://gitlab.kitware.com/cmake/cmake/merge_requests/1632
> The issue that's blocking it now is that the kwsysProcess framework
> isn't thread safe. As a consequence it is not possible for threads to
> start processes concurrently using the kwsysProcess framework.
> 
> There are three options I see
> 
> 1) Make kwsysProcess thread safe
> 
> I don't think it's impossible but it's not a small task.
> Looking through the code I found a few issues that
> would block the whole idea for me.
>  - OpenVMS: I have little to no knowledge about OpenVMS.
>Wikipedia says it is basically abandoned.
>Can the OpenVMS code be removed? I couldn't test it anyway.
> 
>  - C vs. C++
>There's a lot of C in processUNIX.c (list allocation etc.).
>I would prefer to use C++11 (or higher), especially
>std::array, std::vector, std::thread,  std::mutex, etc..
>Is C++11 acceptable in kwsysProcess?
>This would make processUNIX.c processUNIX.cpp?
> 
> 2) Use libuv instead
> 
> Using the libuv event loop is overkill and would probably imply that
> more threads are started than necessary
> (the libuv thread pool will be started anyway).
> But it would allow to start multiple concurrent  (moc/uic) processes.
> Is there a reason to not use libuv?
> 
> 3) Abandon the idea to parallelize AUTOMOC/UIC
> 
> Well, I liked the idea.
> 
> 
> Any thoughts?
> 
> -Sebastian
> 
> --
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake-developers
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

On Freitag, 5. Januar 2018 13:00:30 CET clin...@elemtech.com wrote:
> Cool.  I'd like to have parallel automoc and uic.
> 
> How about this 4th option?
> Don't use threads.  A single thread is able to spawn multiple processes, and
> wait on multiple processes, and react when 1 or more processes change
> state.  I'm probably not familiar enough with kwsysProcess to know if this
> approach will work, but it might be easier and simpler(?) than making it
> thread safe.
> 
> Clint
> 

AFAIK there are two interfaces inside CMake that allow to start a process,
kwsysProcess and libuv.

kwsysProcess allows to start a single process and wait for it to finish. I 
thinks it's also possible to start multiple parallel processes.
But there's no interface to select/poll a number of parallel processes.

So the 4th option would boil down to use libuv for process management.

Sebastian

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

[cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

Hello!

As you might have noticed I tried to parallelize AUTOMOC/UIC.
https://gitlab.kitware.com/cmake/cmake/merge_requests/1632
The issue that's blocking it now is that the kwsysProcess framework
isn't thread safe. As a consequence it is not possible for threads to
start processes concurrently using the kwsysProcess framework.

There are three options I see

1) Make kwsysProcess thread safe

I don't think it's impossible but it's not a small task.
Looking through the code I found a few issues that
would block the whole idea for me.
  - OpenVMS: I have little to no knowledge about OpenVMS. 
Wikipedia says it is basically abandoned. 
Can the OpenVMS code be removed? I couldn't test it anyway.

  - C vs. C++
There's a lot of C in processUNIX.c (list allocation etc.).
I would prefer to use C++11 (or higher), especially
std::array, std::vector, std::thread,  std::mutex, etc..
Is C++11 acceptable in kwsysProcess?
This would make processUNIX.c processUNIX.cpp?

2) Use libuv instead

Using the libuv event loop is overkill and would probably imply that
more threads are started than necessary
(the libuv thread pool will be started anyway).
But it would allow to start multiple concurrent  (moc/uic) processes.
Is there a reason to not use libuv?

3) Abandon the idea to parallelize AUTOMOC/UIC

Well, I liked the idea.


Any thoughts?

-Sebastian
 
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers