Re: [cmake-developers] Faster automoc with parallel processing?
Hi Alexander, Thanks for replying. On 2012-06-18 21:58, Alexander Neundorf wrote: On Sunday 17 June 2012, Thomas Sondergaard wrote: On Windows with MSVC 2010 Generator, moc seems to be very slow and it is a paint to see it run sequentially. Could automoc be taught to do this in parallel using all the cores on the machine? is certainly good enough as an answer ? ;-) It's a good start :-) The thing is, for each target there is an accompanying automoc target. So if you have 4 cores, and 4 targets are built in parallel, then also 4 (auto)mocs should be running in parallel. If you have only 1 target, consisting of many files, only 1 automoc target will be executed, processing one moc after the other. Interesting. I haven't actually fully switched to automoc, I have only tested it on one target in a project with +50 targets. Perhaps the situation will look better if I automoc everything. Parallelizing this is not obvious. Simply running as many mocs on a machine as it has cores might lead to 16 mocs running on 4 cores (for 4 different targets). I think overparallization is better than underparallization. I don't think running 16 parallel mocs at a time on a 4 core machine should be dramatically slower than running 4. An experiment could show. So, it is possible, it is some implementation work, and I'm not clear on how to do the logic. Do you have some ideas how to do this in a good way ? automoc could start a background process and use that to handle all moc'ing. The process could stop after a few seconds of inactivity. That way moc would never use more than the number of cores in the machine, but it might overlap with other build activity. It doesn't seem sensible that automoc should only be concerned about overutilization from other mocs. Perhaps the best way is simply to look at the load-average, like make --load-average? Regards, Thomas -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
[cmake-developers] [CMake 0013316]: FIND_PACKAGE ( X11 REQUIRED ) returns garbage...
The following issue has been SUBMITTED.
==
http://public.kitware.com/Bug/view.php?id=13316
==
Reported By:George Petasis
Assigned To:
==
Project:CMake
Issue ID: 13316
Category: CMake
Reproducibility:always
Severity: block
Priority: normal
Status: new
==
Date Submitted: 2012-06-19 05:29 EDT
Last Modified: 2012-06-19 05:29 EDT
==
Summary:FIND_PACKAGE ( X11 REQUIRED ) returns garbage...
Description:
Hi all,
The following code:
MESSAGE ( STATUS Searching for X11... )
FIND_PACKAGE ( X11 REQUIRED )
IF ( X11_FOUND )
INCLUDE_DIRECTORIES ( ${X11_INCLUDE_DIR} )
LINK_LIBRARIES ( ${X11_LIBRARIES} )
MESSAGE ( STATUS X11_INCLUDE_DIR: ${X11_INCLUDE_DIR} )
MESSAGE ( STATUS X11_LIBRARIES:${X11_LIBRARIES} )
ENDIF ( X11_FOUND )
Prints under Fedora 17:
Searching for X11...
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Found X11: /usr/lib/libX11.so
-- X11_INCLUDE_DIR:
/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include/usr/include
-- X11_LIBRARIES:
/usr/lib/libSM.so/usr/lib/libICE.so/usr/lib/libX11.so/usr/lib/libXext.so
==
Issue History
Date ModifiedUsername FieldChange
==
2012-06-19 05:29 George Petasis New Issue
==
--
Powered by www.kitware.com
Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
[cmake-developers] [CMake 0013317]: Generated moc files are not cleaned
The following issue has been SUBMITTED. == http://www.cmake.org/Bug/view.php?id=13317 == Reported By:Rolf Eike Beer Assigned To:Alex Neundorf == Project:CMake Issue ID: 13317 Category: CMake Reproducibility:always Severity: minor Priority: normal Status: assigned == Date Submitted: 2012-06-19 05:59 EDT Last Modified: 2012-06-19 05:59 EDT == Summary:Generated moc files are not cleaned Description: After make clean the automatically generated moc_*.cpp files are still around. == Issue History Date ModifiedUsername FieldChange == 2012-06-19 05:59 Rolf Eike Beer New Issue 2012-06-19 05:59 Rolf Eike Beer Status new = assigned 2012-06-19 05:59 Rolf Eike Beer Assigned To = Alex Neundorf == -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] Not in next branches on the CMake stage...
On Di., 19. Jun. 2012 19:53:39 CEST, David Cole david.c...@kitware.com wrote: Alex and Eike, The following topic branches have been on the CMake stage for months, without any movement. Are there further plans to move these topics forward and get them into 'next' or are they simply abandoned at this point...? Shouldn't there be a list here? Eike -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] Not in next branches on the CMake stage...
On Tue, Jun 19, 2012 at 2:04 PM, Rolf Eike Beer e...@sf-mail.de wrote: On Di., 19. Jun. 2012 19:53:39 CEST, David Cole david.c...@kitware.com wrote: Alex and Eike, The following topic branches have been on the CMake stage for months, without any movement. Are there further plans to move these topics forward and get them into 'next' or are they simply abandoned at this point...? Shouldn't there be a list here? Eike Uh, why yes, of course. Perhaps since it's after lunch now, I should consider pouring myself a second cup of coffee for the day. (A call to 'ssh g...@cmake.org stage cmake print' shows next=0 for those branches not in 'next' at any given moment) # Not in 'next': # debug-messages | master=0 next=0 #lib64-cleanup | master=0 next=0 # NinjaEclipse | master=0 next=0 Thx, David -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] Not in next branches on the CMake stage...
(A call to 'ssh g...@cmake.org stage cmake print' shows next=0 for those branches not in 'next' at any given moment) # Not in 'next': # debug-messages | master=0 next=0 We didn't reach consensus about that, so I removed it. #lib64-cleanup | master=0 next=0 I've removed that too, but not entirely. I've split out the things that I see as uncritical and but them in their own branches and merged to next. The other cleanups I'll push again after 2.8.9. Eike -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] Not in next branches on the CMake stage...
On Tuesday 19 June 2012, David Cole wrote: On Tue, Jun 19, 2012 at 2:04 PM, Rolf Eike Beer e...@sf-mail.de wrote: On Di., 19. Jun. 2012 19:53:39 CEST, David Cole david.c...@kitware.com wrote: Alex and Eike, The following topic branches have been on the CMake stage for months, without any movement. Are there further plans to move these topics forward and get them into 'next' or are they simply abandoned at this point...? Shouldn't there be a list here? Eike Uh, why yes, of course. Perhaps since it's after lunch now, I should consider pouring myself a second cup of coffee for the day. (A call to 'ssh g...@cmake.org stage cmake print' shows next=0 for those branches not in 'next' at any given moment) # Not in 'next': ... # NinjaEclipse | master=0 next=0 At some point I want to get this working. I'm actually not convinced I'll continue on what I have there, or whether to start fresh. If it's a problem for you, feel free to remove it. Alex -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] Not in next branches on the CMake stage...
On Tue, Jun 19, 2012 at 3:58 PM, Alexander Neundorf neund...@kde.orgwrote: ** On Tuesday 19 June 2012, David Cole wrote: On Tue, Jun 19, 2012 at 2:04 PM, Rolf Eike Beer e...@sf-mail.de wrote: On Di., 19. Jun. 2012 19:53:39 CEST, David Cole david.c...@kitware.com wrote: Alex and Eike, The following topic branches have been on the CMake stage for months, without any movement. Are there further plans to move these topics forward and get them into 'next' or are they simply abandoned at this point...? Shouldn't there be a list here? Eike Uh, why yes, of course. Perhaps since it's after lunch now, I should consider pouring myself a second cup of coffee for the day. (A call to 'ssh g...@cmake.org stage cmake print' shows next=0 for those branches not in 'next' at any given moment) # Not in 'next': ... # NinjaEclipse | master=0 next=0 At some point I want to get this working. I'm actually not convinced I'll continue on what I have there, or whether to start fresh. If it's a problem for you, feel free to remove it. Alex Not really a problem -- just asking about them to find out the intended eventual fate of each branch. Thanks for your responses. D -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
[cmake-developers] [CMake 0013320]: UINT32_C() et al. macro shenanigans break build on HP-UX
The following issue has been SUBMITTED. == http://public.kitware.com/Bug/view.php?id=13320 == Reported By:Daniel R. Gomez Assigned To: == Project:CMake Issue ID: 13320 Category: CMake Reproducibility:always Severity: minor Priority: normal Status: new == Date Submitted: 2012-06-19 16:08 EDT Last Modified: 2012-06-19 16:08 EDT == Summary:UINT32_C() et al. macro shenanigans break build on HP-UX Description: Bootstrapping CMake 2.8.8 on this HP-UX system fails with [ 1%] Building C object Utilities/KWIML/test/CMakeFiles/cmIML_test.dir/test_INT_C.c.o cd /tmp/cmake-build/Utilities/KWIML/test /opt/tg/bin/tg-gcc -DKWIML_NAMESPACE=cmIML -DKWIML_LANGUAGE_C -DKWIML_LANGUAGE_CXX -pipe -W -Wall -Wcast-align -Wformat=2 -Wpointer-arith -Wundef -mpa-risc-2-0 -O3 -Wno-format -I/tmp/cmake-build/Utilities/KWIML/test -I/tmp/cmake-build/Utilities-o CMakeFiles/cmIML_test.dir/test_INT_C.c.o -c /home/src/cmake-2.8.8/Utilities/KWIML/test/test_INT_C.c In file included from /home/src/cmake-2.8.8/Utilities/KWIML/test/test_INT_C.c:14:0: /tmp/cmake-build/Utilities/KWIML/test/test_INT_format.h: In function 'test_INT_format': /tmp/cmake-build/Utilities/KWIML/test/test_INT_format.h:195:1: error: pasting ) and l does not give a valid preprocessing token /tmp/cmake-build/Utilities/KWIML/test/test_INT_format.h:195:3: error: expected ',' or ';' before 'l' gmake[2]: *** [Utilities/KWIML/test/CMakeFiles/cmIML_test.dir/test_INT_C.c.o] Error 1 gmake[2]: Leaving directory `/tmp/cmake-build' gmake[1]: *** [Utilities/KWIML/test/CMakeFiles/cmIML_test.dir/all] Error 2 gmake[1]: Leaving directory `/tmp/cmake-build' gmake: *** [all] Error 2 The test_INT_format.h makes use of a symbol UINT32_C, and others like it. This is defined in inttypes.h: $ gnu-grep -R UINT32_C /usr/include /usr/include/inttypes.h:#define UINT32_C(__c) __CONCAT_U__(__c) /usr/include/inttypes.h:#define UINT32_C(__c) __CONCAT__(__CONCAT_U__(__c),l) /usr/include/inttypes.h:# define UINT32_MAX UINT32_C(4294967295) From what I can tell, this macro is being expanded before being passed as an argument to TEST_C(). I'm not sure how to work around this. == Issue History Date ModifiedUsername FieldChange == 2012-06-19 16:08 Daniel R. GomezNew Issue == -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] Faster automoc with parallel processing?
On Tuesday 19 June 2012, Thomas Sondergaard wrote: Hi Alexander, Thanks for replying. On 2012-06-18 21:58, Alexander Neundorf wrote: On Sunday 17 June 2012, Thomas Sondergaard wrote: On Windows with MSVC 2010 Generator, moc seems to be very slow and it is a paint to see it run sequentially. Could automoc be taught to do this in parallel using all the cores on the machine? is certainly good enough as an answer ? ;-) It's a good start :-) The thing is, for each target there is an accompanying automoc target. So if you have 4 cores, and 4 targets are built in parallel, then also 4 (auto)mocs should be running in parallel. If you have only 1 target, consisting of many files, only 1 automoc target will be executed, processing one moc after the other. Interesting. I haven't actually fully switched to automoc, I have only tested it on one target in a project with +50 targets. Perhaps the situation will look better if I automoc everything. Parallelizing this is not obvious. Simply running as many mocs on a machine as it has cores might lead to 16 mocs running on 4 cores (for 4 different targets). I think overparallization is better than underparallization. I don't think running 16 parallel mocs at a time on a 4 core machine should be dramatically slower than running 4. An experiment could show. Considering a 8 or 16 core machine, this could give already 256 mocs running on 16 cores. I think this would then really be a bit much. I'd actually have to check first whether most of the time is spent in executing moc, or in parsing the files for stuff to be moc'ed. Also the parsing can still be optimized by caching results from earlier runs (which it doesn't do right now). For executing moc, this could also be limited to some number, like 2 or 4, which should already bring a lot. Anyway, currently executing moc is very simple, it's a system-style call, i.e. it starts the executable and blocks until the process has exited. Making this non-blocking is a bit of work. So, it is possible, it is some implementation work, and I'm not clear on how to do the logic. Do you have some ideas how to do this in a good way ? automoc could start a background process and use that to handle all moc'ing. The process could stop after a few seconds of inactivity. That way moc would never use more than the number of cores in the machine, but it might overlap with other build activity. It doesn't seem sensible that automoc should only be concerned about overutilization from other mocs. Perhaps the best way is simply to look at the load-average, like make --load-average? This must be cross-platform, i.e. work on Windows, OSX, Linux, *BSD, AIX, etc. This would be quite an effort I think. Alex -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] Faster automoc with parallel processing?
On 2012-06-19 22:13, Alexander Neundorf wrote: Considering a 8 or 16 core machine, this could give already 256 mocs running on 16 cores. I think this would then really be a bit much. I'd actually have to check first whether most of the time is spent in executing moc, or in parsing the files for stuff to be moc'ed. Also the parsing can still be optimized by caching results from earlier runs (which it doesn't do right now). For executing moc, this could also be limited to some number, like 2 or 4, which should already bring a lot. Anyway, currently executing moc is very simple, it's a system-style call, i.e. it starts the executable and blocks until the process has exited. Making this non-blocking is a bit of work. Right, as it turns out after I have automoc'ed all our projects I am getting better parallelity. As you can see on this image I am getting good value for my four cores and an incredibuild of our complete package is down from about 12 minutes to 8 minutes and 30 seconds. http://spear.medical-insight.com/~ts/automoc_build_time.png So a big thanks to you and the CMake crew for giving us automoc support. automoc could start a background process and use that to handle all moc'ing. The process could stop after a few seconds of inactivity. That way moc would never use more than the number of cores in the machine, but it might overlap with other build activity. It doesn't seem sensible that automoc should only be concerned about overutilization from other mocs. Perhaps the best way is simply to look at the load-average, like make --load-average? This must be cross-platform, i.e. work on Windows, OSX, Linux, *BSD, AIX, etc. This would be quite an effort I think. I agree, before spending time on more parallelization it would be interesting to take a look at moc itself. It appears to be significantly slower on Windows than on Linux. Thanks again, Thomas -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] workflow
On 06/19/2012 04:09 PM, Peter Kümmel wrote: Some small questions to the workflow: - I read on the workflow description site Topic Branch ... Heads not published (no named branch on server) What does this mean? I see all the named branches. I don't know to what text you refer. Where did you read this? - And when I start browsing at http://cmake.org/gitweb the stage repository is not listed, is this by accident? We intentionally left it out to avoid confusing people looking for the CMake repo. Developers get the link to the stage from the Wiki: http://www.cmake.org/Wiki/CMake/Git#Repositories - Is cmake.git automatically synchronized with stage/master and stage/next? Yes. Actually it is the other way: stage/(master|next) are fetched from origin/(master|next). Scripts running in the stage repo construct the merges but do not update the branches locally. Instead the merges must be pushed to origin and then fetched back from there. That avoids races between pushes to the two repos. The general topic stage workflow is documented here: http://www.cmake.org/Wiki/Git/Workflow/Stage with ascii-art history diagrams at each step. -Brad -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
[cmake-developers] [CMake 0013321]: ABI.h doesn't know about older HP compilers
The following issue has been SUBMITTED. == http://public.kitware.com/Bug/view.php?id=13321 == Reported By:Daniel R. Gomez Assigned To: == Project:CMake Issue ID: 13321 Category: CMake Reproducibility:always Severity: minor Priority: normal Status: new == Date Submitted: 2012-06-20 01:55 EDT Last Modified: 2012-06-20 01:55 EDT == Summary:ABI.h doesn't know about older HP compilers Description: Bootstrapping CMake 2.8.8 on HP-UX 11.00 with the vendor-provided C compiler fails with [ 0%] Building C object Utilities/KWIML/test/CMakeFiles/cmIML_test.dir/test_ABI_C.c.o cd /tmp/cmake-2.8.8/_build/Utilities/KWIML/test /opt/ansic/bin/cc -DKWIML_NAMESPACE=cmIML -DKWIML_LANGUAGE_C -DKWIML_LANGUAGE_CXX -Aa -Ae -Ae +DA2.0 +ESlit +w1 -z -Wp,-H262144 +O2 -I/tmp/cmake-2.8.8/_build/Utilities/KWIML/test -I/tmp/cmake-2.8.8/_build/Utilities-o CMakeFiles/cmIML_test.dir/test_ABI_C.c.o -c /tmp/cmake-2.8.8/Utilities/KWIML/test/test_ABI_C.c cpp: /tmp/cmake-2.8.8/_build/Utilities/cmIML/ABI.h, line 163: error 4062: Signedness of 'char' unknown. cpp: /tmp/cmake-2.8.8/_build/Utilities/cmIML/ABI.h, line 257: error 4062: Existence of 'long long' unknown. *** Error exit code 1 This compiler does not define __HP_cc nor __HP_aCC. I think it may be necessary to do the equivalent of #if defined(__hpux) \ !defined(__HP_cc) \ !defined(__HP_aCC) \ !defined(__GNUC__) in order to catch this case. (__hpux is specifically mentioned in the cpp(1) man page.) Version information on this compiler, for reference: $ cc -V -c foo.c cpp.ansi: HP92453-01 A.11.01.20 HP C Preprocessor (ANSI) ccom: HP92453-01 A.11.01.20 HP C Compiler == Issue History Date ModifiedUsername FieldChange == 2012-06-20 01:55 Daniel R. GomezNew Issue == -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
