Re: [cmake-developers] kwsysProcess threadsafety

[Alexander Neundorf]

On 2018 M01 8, Mon 10:32:22 CET Jean-Michaël Celerier wrote:
> > 3) Abandon the idea to parallelize AUTOMOC/UIC
> 
> please no ! moc is a huge bottleneck in my builds (to the point that using
> verdigris  instead ends up having
> faster build time overall).

you could also just not use automoc and use the "manual" moc-wrapper macros. 
This should make the build-time overhead go away.
Instead the developer has to add the header file to be moc'ed to the 
CMakeLists.txt (personally I consider this acceptable).

Alex

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Brad King]

On 01/08/2018 07:26 AM, Sebastian Holtermann wrote:
> Then I'm going to refactor cmQtAutoGeneratorMocUic
> to use a libuv event loop internally.

Great.  Take a look at Source/cmUVHandlePtr.h for some C++ helpers.

-Brad
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Sebastian Holtermann]

On Montag, 8. Januar 2018 06:56:57 CET Brad King wrote:
> On 01/06/2018 02:31 AM, Sebastian Holtermann wrote:
> >>> 2) Use libuv instead
> > 
> > I just saw the libuv library in the CMake sources.
> > 
> >> libuv for process management is on the list. I think it is waiting for
> >> porting to all of CMake's platforms to actually happen.
> > 
> > Does that mean it's only there for some specific scenarios and shouldn't
> > be used in general?
> 
> A goal is to eventually fully replace use of KWSys Process with libuv.
> I'd prefer not to make any major changes to KWSys Process in the meantime.
> 
> Currently libuv is used only for the server mode, but soon it will be
> used for CTest too:
> 
>   https://gitlab.kitware.com/cmake/cmake/merge_requests/1455
> 
> > Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child
> > processes are running and to get a notification when one has finished.
> 
> Use of libuv's process management for that would be good.  It supports
> fully asynchronous operation and so can manage many child processes from
> one thread.

Okay.
Then I'm going to refactor cmQtAutoGeneratorMocUic
to use a libuv event loop internally.

-Sebastian

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Brad King]

On 01/06/2018 02:31 AM, Sebastian Holtermann wrote:
>>> 2) Use libuv instead
> 
> I just saw the libuv library in the CMake sources.
>  
>> libuv for process management is on the list. I think it is waiting for
>> porting to all of CMake's platforms to actually happen.
> 
> Does that mean it's only there for some specific scenarios and shouldn't
> be used in general?

A goal is to eventually fully replace use of KWSys Process with libuv.
I'd prefer not to make any major changes to KWSys Process in the meantime.

Currently libuv is used only for the server mode, but soon it will be
used for CTest too:

  https://gitlab.kitware.com/cmake/cmake/merge_requests/1455

> Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child
> processes are running and to get a notification when one has finished.

Use of libuv's process management for that would be good.  It supports
fully asynchronous operation and so can manage many child processes from
one thread.

Thanks,
-Brad
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] kwsysProcess threadsafety

[Jean-Michaël Celerier]

> 3) Abandon the idea to parallelize AUTOMOC/UIC

please no ! moc is a huge bottleneck in my builds (to the point that using
verdigris  instead ends up having
faster build time overall).

Best,
Jean-Michaël



---
Jean-Michaël Celerier
http://www.jcelerier.name

On Fri, Jan 5, 2018 at 8:34 PM, Sebastian Holtermann 
wrote:

> Hello!
>
> As you might have noticed I tried to parallelize AUTOMOC/UIC.
> https://gitlab.kitware.com/cmake/cmake/merge_requests/1632
> The issue that's blocking it now is that the kwsysProcess framework
> isn't thread safe. As a consequence it is not possible for threads to
> start processes concurrently using the kwsysProcess framework.
>
> There are three options I see
>
> 1) Make kwsysProcess thread safe
>
> I don't think it's impossible but it's not a small task.
> Looking through the code I found a few issues that
> would block the whole idea for me.
>   - OpenVMS: I have little to no knowledge about OpenVMS.
> Wikipedia says it is basically abandoned.
> Can the OpenVMS code be removed? I couldn't test it anyway.
>
>   - C vs. C++
> There's a lot of C in processUNIX.c (list allocation etc.).
> I would prefer to use C++11 (or higher), especially
> std::array, std::vector, std::thread,  std::mutex, etc..
> Is C++11 acceptable in kwsysProcess?
> This would make processUNIX.c processUNIX.cpp?
>
> 2) Use libuv instead
>
> Using the libuv event loop is overkill and would probably imply that
> more threads are started than necessary
> (the libuv thread pool will be started anyway).
> But it would allow to start multiple concurrent  (moc/uic) processes.
> Is there a reason to not use libuv?
>
> 3) Abandon the idea to parallelize AUTOMOC/UIC
>
> Well, I liked the idea.
>
>
> Any thoughts?
>
> -Sebastian
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake-developers
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers