Re: [CMake] How to work around broken Find module
On Thursday 28 October 2010, Marcel Loose wrote: Hi all, What is the best way to work around a broken Find module? A technique I've used up till now is to wrap the broken module in a module with the same name that I store in my own module directory. This way, my wrapper module will be picked up first. Yes. Problem with this approach is, of course, that this might break when someone is using a newer CMake in which the broken module has been fixed. Where is the breakage there ? Your module, which does what you want, still overrides the one coming from cmake, no matter whether that one is better or worse. We don't have a problem with this in KDE, where we have many modules. Do you have a specific problem with some module currently ? If so, which one, what is the exact problem ? A different approach would be to copy the fixed module from the newer CMake and store it in my own module directory. But this might also break, when newer versions of CMake are released. How does that differ from using a module which you modified ? It is still a module which is preferred over the ones shipped with CMake. Alex ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
Hi Andreas! Our build is taking ages (almost a three hours on the fastest of our servers) and it would be really painful if everybody needed to rebuild everything for himself in the morning. 3 hours sounds quite excessively long. Yepp, that's too long, really. Thanks for your suggestions on cutting that down. We have tried a lot of them already, but there simply is no short-term solution to make the separation of the modules clearer. Always working on adding features, never time to refactor, meta template programming all over the place. Same old, same old... Our windows release build and packaging takes almost 8 hours (on a virtual machine), which makes a quick fix of a bug reported by our test team effectively impossible. Using parallel compilation on multiple cores and/or machines would make things easier for us. With linux, we are using distcc and make -jX. The trouble with that is, that the Makefiles generated by qmake don't deal well with generated code which we also use a lot. You have to restart make a few times to keep things going, which is not practical for an automated build and package script. Windows and nmake don't offer any feasible way to parallelize, so we are stuck there. Maybe cmakes VS-Project output could improve the latter and cmake custom command the former. According to the FAQ, CMake does not support copying build trees around. Yep, and I'd think that that's a pretty hard condition. One really shouldn't try to bend things to work with specific build tree data on multiple machines I'd think. That's just not the way that CMake is supposed to be used. Understood, but we do not copy the tree to different machines, just from /home/nightly to /home/developer1, /home/developer2 etc on the same machine. Anyway, thanks for your impression on the situation! Cheers, Benjamin ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
Hi Alan! Our build is taking ages (almost a three hours on the fastest of our servers) and it would be really painful if everybody needed to rebuild everything for himself in the morning. Aren't you distributing your source code with some tool that preserves the creation dates of source files? I think all of svn, rsync, tar, or even cp -a do this. Yes, we do use cvs and copy build trees with rsync -a/cp -a. The problem is that because our builds take that long, nobody does a global cvs update very often, but rather only on the parts he is working on. It's user nightly that does an automated fresh checkout every night and sees if everything is fitting together and passes the tests. That makes it more attractive to copy the nightly build to continue your own work, because there is some guarantee that it builds, passes the tests and incorporated the stuff from the other teams. Cheers, Benjamin ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
On Sunday 31 October 2010, Benjamin King wrote: Hi Andreas! Our build is taking ages (almost a three hours on the fastest of our servers) and it would be really painful if everybody needed to rebuild everything for himself in the morning. 3 hours sounds quite excessively long. Yepp, that's too long, really. Thanks for your suggestions on cutting that down. We have tried a lot of them already, but there simply is no short-term solution to make the separation of the modules clearer. Always working on adding features, never time to refactor, meta template programming all over the place. Same old, same old... Our windows release build and packaging takes almost 8 hours (on a virtual machine), which makes a quick fix of a bug reported by our test team effectively impossible. Using parallel compilation on multiple cores and/or machines would make things easier for us. With linux, we are using distcc and make -jX. The trouble with that is, that the Makefiles generated by qmake don't deal well with generated code which we also use a lot. You have to restart make a few times to keep things going, which is not practical for an automated build and package script. This is handled properly by cmake-generated makefiles and project files. You use add_custom_command() to generate files, and if you really list all files the custom command generates after the OUTPUT keyword, parallel builds will work properly. We do that all the time in KDE. Windows and nmake don't offer any feasible way to parallelize, so we are stuck there. You could use GNU make with the MS compiler, or you can use jom, which is a nmake compatible make for Windows, which supports parallel builds (http://labs.qt.nokia.com/2009/03/27/speeding-up-visual-c-qt-builds/) Alex ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
On 2010-10-31 13:48+0100 Benjamin King wrote: Hi Alan! Our build is taking ages (almost a three hours on the fastest of our servers) and it would be really painful if everybody needed to rebuild everything for himself in the morning. Aren't you distributing your source code with some tool that preserves the creation dates of source files? I think all of svn, rsync, tar, or even cp -a do this. Yes, we do use cvs and copy build trees with rsync -a/cp -a. The problem is that because our builds take that long, nobody does a global cvs update very often, but rather only on the parts he is working on. It's user nightly that does an automated fresh checkout every night and sees if everything is fitting together and passes the tests. That makes it more attractive to copy the nightly build to continue your own work, because there is some guarantee that it builds, passes the tests and incorporated the stuff from the other teams. Hi Benjamin: This sounds like you are describing dependency problems with your current build system. That should be a prime motivation to implement a complete CMake-based build system which allows you to specify all dependencies between your software components with no gratuitious rebuilding. With a properly implemented complete CMake-based build system, your users would be controlling what gets rebuilt in a flexible way depending on what they choose to cvs update. That is the ideal result and therefore a great selling point for CMake-based build systems. Alan __ Alan W. Irwin Astronomical research affiliation with Department of Physics and Astronomy, University of Victoria (astrowww.phys.uvic.ca). Programming affiliations with the FreeEOS equation-of-state implementation for stellar interiors (freeeos.sf.net); PLplot scientific plotting software package (plplot.org); the libLASi project (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net); and the Linux Brochure Project (lbproject.sf.net). __ Linux-powered Science __ ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
On Sun, Oct 31, 2010 at 10:42 AM, Benjamin King benjamink...@web.de wrote: Hi Andreas! Our build is taking ages (almost a three hours on the fastest of our servers) and it would be really painful if everybody needed to rebuild everything for himself in the morning. 3 hours sounds quite excessively long. Yepp, that's too long, really. Thanks for your suggestions on cutting that down. We have tried a lot of them already, but there simply is no short-term solution to make the separation of the modules clearer. Always working on adding features, never time to refactor, meta template programming all over the place. Same old, same old... Our windows release build and packaging takes almost 8 hours (on a virtual machine), which makes a quick fix of a bug reported by our test team effectively impossible. Using parallel compilation on multiple cores and/or machines would make things easier for us. With linux, we are using distcc and make -jX. The trouble with that is, that the Makefiles generated by qmake don't deal well with generated code which we also use a lot. You have to restart make a few times to keep things going, which is not practical for an automated build and package script. Windows and nmake don't offer any feasible way to parallelize, so we are stuck there. If you haven't already, you might want to take a lot at JOM, which is essentially parallel nmake: http://qt.gitorious.org/qt-labs/jom. JOM is supported by CMake. Maybe cmakes VS-Project output could improve the latter and cmake custom command the former. According to the FAQ, CMake does not support copying build trees around. Yep, and I'd think that that's a pretty hard condition. One really shouldn't try to bend things to work with specific build tree data on multiple machines I'd think. That's just not the way that CMake is supposed to be used. Understood, but we do not copy the tree to different machines, just from /home/nightly to /home/developer1, /home/developer2 etc on the same machine. Anyway, thanks for your impression on the situation! Cheers, Benjamin ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] CMake 2.8.3-rc4 ready for testing!
Hi I saw in the log you added a case for Python 2.7, Would you be able to add a check for Python 3.x ? For Blender 3D we use CMake and only support python 3.x series. ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] CMake 2.8.3-rc4 ready for testing!
On Sun, Oct 31, 2010 at 7:30 PM, Campbell Barton ideasma...@gmail.com wrote: Hi I saw in the log you added a case for Python 2.7, Would you be able to add a check for Python 3.x ? For Blender 3D we use CMake and only support python 3.x series. I added that (or part of it at least). I suspect it is too late, and we really need to add a variable to request Python 3. That would be too big of a change this late in the RC series. We could work on something, with a view to getting it in CMake 2.8.4, that you could add to Blender's CMake module path until the release. I think for it to be general enough we would need something that defaulted to Python 2.x, but could look for Python 3 if the project requested it. I was not sure how best to handle this, and wanted to make sure this release found Python 2.7. I know for most of the project I develop we explicitly don't want Python 3 as our stuff is not ready for it yet. Marcus ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
Am 30.10.2010 13:54, schrieb Benjamin King: Hello, I'm working on a ~1.5Mio LOC C++ project and our buildsystem is a hodgepodge of handcrafted Makefiles, shell scripts and qmake projects. I tried to convert a subset to CMake and it looks very promising so far. I'm currently (as a consultant) converting ~2.5 Mio LOC C++/C from vcproj/Makefiles/qmake to cmake. Working with big projects, different VCS, lots of developers is very different to using cmake in small projects (few developers, single repository). Here are some things I've learned so far: - Split repositories (You can not put ~2.5 Mio LOC in one repository) - Automate everything (You need: continuous build and delivery, use cdash or hudson) - Complete build process (You need: checkout, build, packaging and install) - Support developers (You need: developers want to work as they used to with their IDE) - Fill gaps (You need: Find*-Modules, CMake-Macros) One important part of our development workflow is this: 1) User 'nightly' builds versions of the project every night on several development servers. 2) A developer is coming to the office in the morning, and copies/hardlinks the nightly build to his home dir and probably patches uncommited changes from yesterday into the newly copied build. Nightly builds are very useful to build releases. Developers should have their own source tree and update it on their own (daily or more often updates from repository and incremental rebuilds). Systems for continuous integration might also be useful. Our build is taking ages (almost a three hours on the fastest of our servers) and it would be really painful if everybody needed to rebuild everything for himself in the morning. According to the FAQ, CMake does not support copying build trees around. On the development servers, we have the special situation that all external tools needed for the build are either in /usr/bin, /usr/local/bin or /opt/local/ourstuff/build/bin. Would it be feasible to copy a build tree in such a setting? Or could we copy everything, throw the CmakeCache away, reconfigure and reuse the object files that already were build? There are others ways to build faster, like ccache (http://ccache.samba.org/) or distcc (http://distcc.samba.org/). Do you have any advice on this or other ideas? This is really a make-or-break feature for us and I'd like to present some solution with CMake for it. If you are going to change tools (introduce cmake), it is also time to review your development process (If in the business world e.g. a SAP-Systems is introduced the complete work flow and business processes will get adapted). Regards, Karl ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
Hello Alex! [Dependencies to generated files are] handled properly by cmake-generated makefiles and project files. You use add_custom_command() to generate files, and if you really list all files the custom command generates after the OUTPUT keyword, parallel builds will work properly. Great, that's good to know. You could use GNU make with the MS compiler, or you can use jom, which is a nmake compatible make for Windows, which supports parallel builds (http://labs.qt.nokia.com/2009/03/27/speeding-up-visual-c-qt-builds/) Thanks again. I'll try both! Cheers, Benjamin ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
Hi Pedro! Windows and nmake don't offer any feasible way to parallelize, so we are stuck there. If you haven't already, you might want to take a lot at JOM, which is essentially parallel nmake: http://qt.gitorious.org/qt-labs/jom. JOM is supported by CMake. Thank you Pedro. One of our developers already tried to plug jom into our current build system, but some trouble with forward slashes in path names occurred. I'll give it a shot with CMake! Cheers, Benjamin ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
Re: [CMake] Converting a large C++-Project to CMake
Hi Karl! I'm currently (as a consultant) converting ~2.5 Mio LOC C++/C from vcproj/Makefiles/qmake to cmake. Can you say something of the motivations for your client to do this? I suspect that you are expected to keep most of their development workflow and processes in order. Which parts are you allowed to redesign and which have to stay? For us, there are also strong non-technical factors that will affect the decision to improve the build system, such as: 1) Is there some migration path from the old system? 2) Can we get around taking two weeks off where everybody needs to learn a new tool? 3) Can we keep the knowledge that went into the scripts, makefiles and qmake-projects of the current system? 4) Does it offer enough advantages to take the risk to change a matured and operative build system (albeit the latter is running at its limits)? There are more points, but I'll need to address them all in order to convince anyone to even try cmake. Right now, my plan is to convert a substantial part of the project to cmake. It needs to work with all our code generators. It will include a subproject that has a substantial amount of unit, integration and regression test that all need to be passed afterwards. And the migration path could be to just check in the CMakeLists.txt into the repository and be grateful that cmake builds out of source. Part of my non-technical review will be the amount of time and pain it took me to do the conversion. The book (Mastering CMake) and the superfriendly and professional mailing lists will surely be big selling points there :-) I'd be really interesting to know what non-technical issues you are facing with your current project! - Automate everything (You need: continuous build and delivery, use cdash or hudson) - Complete build process (You need: checkout, build, packaging and install) We do that with bash scripts and virtual machines. Which is why running cmake must be fully scriptable. I understand that it is if you are passing it the required configuration parameters with cmake -DTHIS_AND_THAT. There are others ways to build faster, like ccache (http://ccache.samba.org/) or distcc (http://distcc.samba.org/). We are using both for regular developer builds in linux. With release builds, we are a little more reluctant and expect everything to build in bearable time from scratch on a single machine. This works, just short of the bearable part. If you are going to change tools (introduce cmake), it is also time to review your development process (If in the business world e.g. a SAP-Systems is introduced the complete work flow and business processes will get adapted). I'll collect all the suggestions from this list and summarize them in a report. Thanks a lot for being so helpful, that's much appreciated! Cheers, Benjamin ___ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
