[cmake-developers] How to handle dependencies of protobuf files ?
Hi, I stumbled upon a problem with protobuf files, I attached a testcase. There is a MyBase.proto, which is "imported" by Complex.proto. If MyBase.proto is modified, protoc is run again in MyBase.proto, but not on Complex.proto, although it should. You can have a look at the attached example. The message MyData (in Complex.proto) has a member MyBase b1. If I rename the message MyBase (in MyBase.proto) e.g. to MyBaseXYZ, then the build fails, because Complex.pb.h was not regenerated, so it still refered to the now not existing class MyBase. Is there already a solution to handle this ? I think to do it properly, there would have to be a dependency scanning for proto files like there is for C/C++ headers. Parsing at the proto-files at cmake time wouldn't be good enough (since editing a proto file doesn't trigger a cmake run). Comments ? Alex protodeps.tar.gz Description: application/compressed-tar -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake-developers
Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules
On 05/15/2018 03:22 AM, Stephen Kelly wrote: > So, the answer for cmake might be that CMake can learn to extract that > stuff, but ignore certain cases like imports within ifdefs. We'd need to do the extraction from already-preprocessed sources. This is how Fortran+Ninja+CMake works. Unfortunately for C++ this will typically require preprocessing twice: once just to extract module dependencies and again to actually compile. With Fortran we compile using the already-preprocessed source but doing that with C++ will break things like Clang's nice handling of macros in diagnostic messages. -Brad -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake-developers
Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules
David Blaikie wrote: >> Nope, scratch that ^ I had thought that was the case, but talking more >> with Richard Smith it seems there's an expectation that modules will be >> somewhere between header and library granularity (obviously some small >> libraries today have one or only a few headers, some (like Qt) have many >> - maybe those on the Qt end might have slightly fewer modules than the >> have headers - but still several modules to one library most likely, by >> the sounds of it) >> >> >> Why? Richard maybe you can answer that? These are the kinds of things I >> was trying to get answers to in the previous post to iso sg2 in the >> google group. I didn't get an answer as definitive as this, so maybe you >> can share the reason behind such a definitive answer? >> > > It's more that the functionality will allow this & just judging by how > people do things today (existing header granularity partly motivated by > the cost of headers that doesn't apply to modules), how they're likely to > do things in the future (I personally would guess people will probably try > to just port their headers to modules - and a few places where there are > circular dependencies in headers or the like they might glob them up into > one module). It seems quite common to have one PCH file per shared library (that's what Qt does for example). What makes you so sure that won't be the case with modules? I'd say that what people will do will be determined by whatever their tools optimize for. If it is necessary to list all used modules on the compile line, people would choose fewer modules. If 'import QtCore' is fast and allows the use of QString and QVariant etc and there is no downside, then that will be the granularity offered by Qt (instead of 'QtCore.QString'). That is also comparable to '#include ' which is possible today. >> I just looked through the commits from Boris, and it seems he made some >> changes relating to -fmodule-file=. That still presupposes that all >> (transitively) used module files are specified on the command line. >> > > Actually I believe the need is only the immediate dependencies - at least > with Clang's implementation. Ok. That's not much better though. It still means editing/generating the buildsystem each time you add an import. I don't think a model with that requirement will gain adoption. >> I was talking about the -fprebuilt-module-path option added by Manman Ren >> in https://reviews.llvm.org/D23125 because that actually relieves the >> user/buildsystem of maintaining a list of all used modules (I hope). >> > > *nod* & as you say, GCC has something similar. Though the build system > probably wants to know about the used modules to do dependency analysis & > rebuilding correctly. Yes, presumably that will work with -MM. > Yeah, thanks for the link - useful to read. There seems to be a slew of activity around modules at the moment. You can read some other reactions here which might have input for your paper: https://www.reddit.com/r/cpp/comments/8jb0nt/what_modules_can_actually_provide_and_what_not/ https://www.reddit.com/r/cpp/comments/8j1edf/really_think_that_the_macro_story_in_modules_is/ I look forward to reading your paper anyway. >> I think clang outputs the definitions in a separate object file, but GCC >> currently doesn't. Perhaps that's a difference that cmake has to account >> for or pass on to the user. >> > > Clang outputs frontend-usable (not object code, but serialized AST usable > for compiling other source code) descriptions of the entire module > (whatever it contains - declarations, definitions, etc) to the .pcm file. > It can then, in a separate step, build an object file from the pcm. I > think GCC produces both of these artifacts in one go - but not in the same > file. Ok, I must have misremembered something. >> Sure. I didn't notice anything from reading, but I also didn't try it >> out. You might need to provide a repo with the module.modulemap/c++ files >> etc that are part of your experiment. Or better, provide something based >> on modules-ts that I can try out. >> > > *nod* I'll see if I can get enough of modules-ts type things working to > provide some examples, but there's some more variance/uncertainty there in > the compiler support, etc. Something working only with clang for example would be a good start. >> I'm guessing that's enough for you to implement what you want as an >> experiment? >> > > OK, so in that case it requires source changes to cmake? *nod* sounds > plausible - I appreciate the pointers. I take it that implies there's not > a way I could hook into those file kinds and filters without changing > cmake? (ie: from within my project's cmake build files, without modifying > a cmake release) There is no way to hook into the system I described without patching CMake. Your custom command approach might be the way to do that if it is the priority. Thanks, Stephen. -- Powered by www.kitware.com Please keep m
Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules
Brad King wrote: > On 05/07/2018 12:01 PM, Stephen Kelly wrote: >> Hopefully Brad or someone else can provide other input from research >> already done. > > I'm not particularly familiar with what compiler writers or the modules > standard specification expects build systems to do w.r.t modules. > However, IIUC at least at one time the expectation was that the module > files would not be installed like headers and are used only within a > local build tree. Are modules even supposed to be first-class entities > in the build system specification that users write? The answer is probably both 'hopefully not' and 'sometimes'. > In the Fortran world users just list all the sources and build systems are > expected to figure it out. CMake has very good support for Fortran > modules. Our Ninja generator has rules to preprocess the translation units > first, then parse the preprocessed output to extract module definitions > and usages, then inject the added dependencies into the build graph, and > then begin compilation of sources ordered by those dependencies (this > requires a custom fork of Ninja pending upstream acceptance). > > Is that what is expected from C++ buildsystems for modules too? Hopefully. However, in some cases, the step of 'extracting module definitions and usages' might be very hard to do. This document is quite concise about that: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1052r0.html So, the answer for cmake might be that CMake can learn to extract that stuff, but ignore certain cases like imports within ifdefs. Maybe CMake could then also provide API for users to specify the usages/dependencies explicitly in those cases. I don't know how convenient that would be (or could be made through design). Thanks, Stephen. -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: https://cmake.org/mailman/listinfo/cmake-developers
