Re: [cmake-developers] cmake build does too much work
On 12/12/2013 11:46 AM, Matthew Woehlke wrote: Where template.hpp changes (testing with 2.8.12.1 shows that touching template.hpp triggers a rebuild with Ninja, but not Unix Makefiles), That sounds like a bug :-). Yes, this is a bug. ninja and make should be the same. -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On 12/11/2013 5:13 PM, Matthew Woehlke wrote: I've been working on a project lately that isn't *that* massively large, but has an unusually high number of library and executable targets. One thing that's been bugging me is that any trivial change in a lower level library causes more than a hundred targets to be relinked, for no good reason I know of. Now, I *do* get that relinking is good if the library ABI changes. However, that's not the case here, and I am wondering if it would be possible for CMake to generate an additional, intermediary step after library linking to somehow export a file representing the ABI of the library (with overwrite checks to not modify the file if the ABI has not changed), and to use *those*, rather than the actual libraries, as the dependencies for targets linking to the libraries. I think this could produce a significant speed-up for incremental builds in some cases, as it would allow the build to short-circuit the relink of many targets when it turns out a library's ABI has not changed. Does this sound like something CMake could/should do? (I'm thinking something like running objdump on the resulting library with suitable arguments and doing a copy_if_different on the output. I guess this would only apply to shared libraries, and probably should be an optional feature.) There is this property: http://www.cmake.org/cmake/help/git-master/variable/CMAKE_LINK_DEPENDS_NO_SHARED.html If you wanted to use that for your project I think you can just run cmake -DCMAKE_LINK_DEPENDS_NO_SHARED=TRUE. The other option sounds interesting but hard to implement in a cross platform manner. -Bill -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On 2013-12-11 17:40, Bill Hoffman wrote: On 12/11/2013 5:13 PM, Matthew Woehlke wrote: I've been working on a project lately that isn't *that* massively large, but has an unusually high number of library and executable targets. One thing that's been bugging me is that any trivial change in a lower level library causes more than a hundred targets to be relinked, for no good reason I know of. Now, I *do* get that relinking is good if the library ABI changes. However, that's not the case here, and I am wondering if it would be possible for CMake to generate an additional, intermediary step after library linking to somehow export a file representing the ABI of the library (with overwrite checks to not modify the file if the ABI has not changed), and to use *those*, rather than the actual libraries, as the dependencies for targets linking to the libraries. I think this could produce a significant speed-up for incremental builds in some cases, as it would allow the build to short-circuit the relink of many targets when it turns out a library's ABI has not changed. Does this sound like something CMake could/should do? (I'm thinking something like running objdump on the resulting library with suitable arguments and doing a copy_if_different on the output. I guess this would only apply to shared libraries, and probably should be an optional feature.) There is this property: http://www.cmake.org/cmake/help/git-master/variable/CMAKE_LINK_DEPENDS_NO_SHARED.html Yes, but that can lead to a broken build if the ABI does change. (Hi, Stephen. I still oppose making the above on by default due to the possibility of broken builds. The difference here is that we are actually doing extra work in order to determine if a relink is necessary of if it is *safe* to skip it. IOW, I still choose correctness over performance; the difference is that the proposed technique would let us have both :-).) The other option sounds interesting but hard to implement in a cross platform manner. Agreed, but even if it only worked on some platforms it could be a significant benefit to them. It's the sort of feature that can just be ignored if not supported with no loss in correctness (only performance). Actually because of its .lib files I wonder if maybe Windows either already doesn't have this problem, or at least would be much easier to provide this feature there. (I'd be mainly interested in Linux.) Hmm, come to think of it, if this were to be implemented, I wonder if it is faster for the ABI stamp to be an actual dump of the ABI, or a checksum (e.g. md5 / sha256) of the ABI dump... The latter would trade the extra cost computing the checksum for I/O; not needing to write the ABI dump to disk at all if it can be read via a pipe, and then only needing to read and possibly write a few bytes for the 'stamp' file. Actually, I suppose the build tool could also do this, but then the benefit is only realized by users of that build tool. (On the other hand, all users of that build tool would realize the benefit, and not just users of CMake.) Still, this now feels worth cross-posting to ninja... -- Matthew -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On Thu, Dec 12, 2013 at 00:00:42 +0100, Stephen Kelly wrote: You opposed a sensible default for CMAKE_LINK_DEPENDS_NO_SHARED: https://www.mail-archive.com/cmake-developers@cmake.org/msg06169.html I continue to consider the default value of that to be a mistake. How would a relink be forced with the default of ON when the ABI does break? Would I need to invoke the targets manually? That sounds painful to me. --Ben -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On 2013-12-11 17:13, Matthew Woehlke wrote: I've been working on a project lately that isn't *that* massively large, but has an unusually high number of library and executable targets. One thing that's been bugging me is that any trivial change in a lower level library causes more than a hundred targets to be relinked, for no good reason I know of. Now, I *do* get that relinking is good if the library ABI changes. However, that's not the case here, and I am wondering if it would be possible for CMake to generate an additional, intermediary step after library linking to somehow export a file representing the ABI of the library (with overwrite checks to not modify the file if the ABI has not changed), and to use *those*, rather than the actual libraries, as the dependencies for targets linking to the libraries. I think this could produce a significant speed-up for incremental builds in some cases, as it would allow the build to short-circuit the relink of many targets when it turns out a library's ABI has not changed. Does this sound like something CMake could/should do? (I'm thinking something like running objdump on the resulting library with suitable arguments and doing a copy_if_different on the output. I guess this would only apply to shared libraries, and probably should be an optional feature.) FYI, the ninja folks mentioned that GYP already does this for Linux/Mac/Win... maybe we (CMake) could borrow their work? -- Matthew -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On Wed, Dec 11, 2013 at 17:13:00 -0500, Matthew Woehlke wrote: Now, I *do* get that relinking is good if the library ABI changes. However, that's not the case here, and I am wondering if it would be possible for CMake to generate an additional, intermediary step after library linking to somehow export a file representing the ABI of the library (with overwrite checks to not modify the file if the ABI has not changed), and to use *those*, rather than the actual libraries, as the dependencies for targets linking to the libraries. I think this could produce a significant speed-up for incremental builds in some cases, as it would allow the build to short-circuit the relink of many targets when it turns out a library's ABI has not changed. While the tool I posted in the other message is a possibility, how would you deal with inline functions changing? Same with template implementations? For example, when a Boost *header* changes, dependencies need to be recompiled to get the new inline code. Which build systems actually have that level of granularity? Does XCode or VS track external dependencies on the file level? I don't *think* make and ninja do so, since after a GCC upgrade, my trees don't all of a sudden do a full rebuild (and after upgrading from Fedora N to N+1, nuking build trees is usually on the menu due to new soversions of libraries shifting around; a simple rebuild misses this). --Ben -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On 2013-12-11 19:21, Ben Boeckel wrote: On Wed, Dec 11, 2013 at 17:13:00 -0500, Matthew Woehlke wrote: Now, I *do* get that relinking is good if the library ABI changes. However, that's not the case here, and I am wondering if it would be possible for CMake to generate an additional, intermediary step after library linking to somehow export a file representing the ABI of the library (with overwrite checks to not modify the file if the ABI has not changed), and to use *those*, rather than the actual libraries, as the dependencies for targets linking to the libraries. I think this could produce a significant speed-up for incremental builds in some cases, as it would allow the build to short-circuit the relink of many targets when it turns out a library's ABI has not changed. While the tool I posted in the other message is a possibility, how would you deal with inline functions changing? Same with template implementations? I don't think this is relevant? In these cases, a header is changing, which will (hopefully) lead to the source files using that header being rebuilt, which will cause the library to relink anyway. (And if the sources *aren't* rebuilt, I don't think relinking will help?) Whether or not the sources of library B have correct dependencies on the headers of library A is, I believe, orthogonal to this problem, which is *only* about link-level dependencies. IOW the only rule affected would be 'libb.so: liba.so'. Which build systems actually have that level of granularity? Does XCode or VS track external dependencies on the file level? I don't *think* make and ninja do so, since after a GCC upgrade, my trees don't all of a sudden do a full rebuild (and after upgrading from Fedora N to N+1, nuking build trees is usually on the menu due to new soversions of libraries shifting around; a simple rebuild misses this). I would expect the behavior for external dependencies would not change; they would either trigger rebuilds or not the same as they do currently. (Since of course we cannot rely on having 'ABI stamp files' for any external libraries...) I'm *mainly* having problems within a single CMake project, which is what this would affect. It might have some downstream effect due to fewer libraries within the project changing to trigger downstream rebuilds, but that would be more incidental. -- Matthew -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On Wed, Dec 11, 2013 at 19:38:21 -0500, Matthew Woehlke wrote: I don't think this is relevant? In these cases, a header is changing, which will (hopefully) lead to the source files using that header being rebuilt, which will cause the library to relink anyway. (And if the sources *aren't* rebuilt, I don't think relinking will help?) I'm concerned about this: (external) (internal) template.hpp -- A.cpp -- header.h ^^ || |\- liba.so | ^ | | \-- B.cpp -- libb.so Where template.hpp changes (testing with 2.8.12.1 shows that touching template.hpp triggers a rebuild with Ninja, but not Unix Makefiles), internal_header.h changes and A.cpp gets recompiled, liba.so relinked and libb.so skipped because the ABI hasn't changed. The problem is that if something inlined from template.hpp is incompatible with what B.cpp has inlined, things need recompiled. I don't know how other tools work, but Unix Makefiles are certainly popular enough that I'd like to see that this cannot cause problems first. Of course, the fix may be to have A.cpp depend on template.hpp directly. I don't know the cost of this in for the build or generate time (but possibly noticable with files which include a lot). I would expect the behavior for external dependencies would not change; they would either trigger rebuilds or not the same as they do currently. (Since of course we cannot rely on having 'ABI stamp files' for any external libraries...) Maybe the problem I was referencing was out-of-scope. I wonder if abi-compliance-checker handles mismatches between inlined symbols (mostly a problem outside of -fvisibility=hidden). --Ben -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
Re: [cmake-developers] cmake build does too much work
On 12/11/2013 6:00 PM, Stephen Kelly wrote: You opposed a sensible default for CMAKE_LINK_DEPENDS_NO_SHARED: https://www.mail-archive.com/cmake-developers@cmake.org/msg06169.html I continue to consider the default value of that to be a mistake. I am on the fence with this one. It is always best to have an incremental build that you can trust. However, I have to think pretty hard to come up with a case where you could change an object size without changing a header file that would cause a relink anyway. I can come up with ways to do it, but you would almost have to try -Bill -- Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Follow this link to subscribe/unsubscribe: http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers
