Re: [cmake-developers] Generating buildsystem metadata from CMake
On 01 Apr 2015, at 00:41, Stephen Kelly steve...@gmail.com wrote: Anton Makeev wrote: The problem with the following format: --- include_directories : [/foo, /opt] compile_definitions : [DEF=\Foo\, OTHER_DEF=1] compile_flags: [ -c ] — Is that it’s incomplete and cannot be used directly: * include directories list misses compiler-defined search paths CMake gathers those during configure time, and we can add them to the metadata too. However appealing it is, I don’t think it’s a good option, since it'll make generation phase way longer, since files and targets may have different command line options and CMake will need to call compiler for all of them. We’d rather do it lazily when times come to process the given file having specialized caches on IDE side. * quote include directories are not distinguishable CMake does not currently generate -iquote as a result of a user writing an include_directories() call. Are you asking for that as an option? That would be an independent feature request. I'd suggest filing it in the tracker. Otherwise, the only way CMake currently would pass such options to the compiler is if the user specified them in CMAKE_CXX_FLAGS or add_compile_options() or similar. Yes, I mean that user may easily have -iquote directly passed as command line option and the IDE need to know that. CMake does not attempt to parse CMAKE_CXX_FLAGS for things which look like include directories. That variable is just an opaque string of data from the point of view of CMake. It is not clear to me whether you are suggesting that CMake should do that. If you do want to suggest that, please file an issue in the tracker. Similarly, CMake does not attempt to analyse the meaning of options specified in add_compile_options(). Again, it's not clear to me whether you are suggesting a new feature for CMake. If you are, please file it in the tracker. No, I don’t suggest CMake should do that, sorry for the confusion. I was just explaining why we need the whole compiler-specific command line, instead of a list of separate flags and include directories - we’ll pass that command line as is (actually with some post-processing) to the compiler to collect the actual defines, header search paths etc. (some examples below) The current proposal (with having separate include_directories, compile_definitions, compile_flags) will force the IDE to assemble the compiler-specific command line itself that is quite prone to errors and will require constant maintenance since additional options may appear (e.g. include_directories may some day have a flag to indicate whether it “” or ) To avoid misunderstandings, please post snippets of CMake code together with a description of what information you want CMake to determine from the snippet and what metadata it should generate. Not sure what you are asking, I hope I made it more clear with previous explanation. * compiler definition do not include compiler-defined(built-in) definitions etc. CMake does not currently gather those during configure, as far as I know. That could be added though. It'd rather not, since the set may be different for the specific files. That is, to have the complete list of include directories and compiler definitions, the IDE will have to call the compiler anyways. And here is where it will need the actual compiler’s command line. Should we had the full list of include directories (split into groups like: , quote, and frameworks) and the list of all compiler defines, we would not need the compiler command-line at all. I think I understand what you're saying. However, it would help if you related what you write to existing CMake features and snippets of CMake code. I assume that you are medium-to-very familiar with CMake features already, given what you are working on? Given the details above, please let me know, if you still need them, I’ll try to collect the specific cases that we support from our test suite. Though I don’t this it’s a way to go for CMake generation, since the compiler should be called for every unique command-line flags and undoubtedly make generation way longer. It's not completely clear to me what you are suggesting here. Maybe you know something I don't know. Can you be clear about what you are suggesting CMake would have to do for each unique command line? I know you are not proposing that CMake should do it (you said the IDE should do it lazily instead), but what action are you actually talking about? What should the IDE do lazily instead of CMake doing it? Invoke the compiler to determine the compiler-defined/built-in include directories and definitions? They will always be the same, right? Or are you talking about the -I- GCC option here and how that clears the built-ins? Please be more specific. We currently reconstruct the command line from the 'build_dir/CMakeFiles/Target1.dir/flags.make’ file
Re: [cmake-developers] Generating buildsystem metadata from CMake
On 01 Apr 2015, at 00:17, Stephen Kelly steve...@gmail.com wrote: Anton Makeev wrote: It comes from CMAKE_CONFIGURATION_TYPES, if project defined such a variable. With default of Debug/Release/RelWithDebInfo/MinSizeRel Ok. So does this mean you have a parser for the existing CMakeCache.txt format already? Yes, at the moment we parse lots of various internal files created by Makefiles generator, and CMakeCache is one of them. Not only do we reed it, we also have the UI that allows user to interact and the cached values. That means we’ll have CMakeCache parser anyways and do not need these values in metadata.json file, as it was mentioned in parallel discussion. Anton Makeev JetBrains http://www.jetbrains.com Develop with pleasure! smime.p7s Description: S/MIME cryptographic signature -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Generating buildsystem metadata from CMake
On 21 Mar 2015, at 09:41, Stephen Kelly steve...@gmail.com wrote: Anton Makeev wrote: The other thing that seems troubling to me is that since file, target, language compiler options are split into different parts of metadata, the IDE need to know exactly how to assemble them back into the compiler’s command line (e.g. what flags go first file’s or language’s), duplicating CMake's logic that may be different from version to version and from compiler to compiler. The exact command line is needed to get the actual and precise defines, include search paths etc. from the compiler. Yes. I previously proposed the lang_compile_command to contain information about how to built it: http://www.steveire.com/cmake-future/manual/cmake-metadata-generation.7.html#optional-properties Thanks for pointing, I missed the purpose of that property. If we are after a compact form, this should work, I guess. However, I think it might be better to generate something similar to what is currently generated in compile-commands.json into cmake-metadata.json. That is, we would generate (in some context) include_directories : [/foo, /opt] compile_definitions : [DEF=\Foo\, OTHER_DEF=1] compile_command: -c -DDEF=\Foo\ -DOTHER_DEF=1 -I/foo -I/opt So, compile_command contains approximately what you can currently get from compile-commands.json. The other properties contain things which are specifically known to be include directories or compile definitions, as javascript arrays. These properties are obviously redundant information, so I wonder if they should be generated at all? Is the compile command I wrote above easy to parse? Or is it sufficiently difficult to parse that this redundant information should be provided? Agree, I don’t see any point in having this redundant info. I’d better not have it to avoid confusion: the include_directories lists is incomplete (it doesn’t contain compiler-defined search paths) and should not be used for any purpose. This would be really helpful indeed, currently, we have to introspect CMakeLists.txt files in order to find the most probably place where new files should be placed (works only in basic cases now). And being able to do so correctly is also crucial for refactoring (e.g. extract class). Given the backtrace, you can navigate up the scope from the most recent frame to get out of any functions, macros or loops. You can then add a target_sources() line directly after that. That algorithm will work for every case (not just basic cases) as far as I can tell and is available with CMake 3.1. OK, it’s definitely better than nothing, so we can start with the target backtraces. * I didn't document the location or directory. I'm not clear on whether it is supposed to be the build location, or the install location(s!), or all of those. It would be useful, though, to have a location of generated files for each target: in case metadata misses some information (and I think it won’t cover every possible need anytime soon), IDE will be able to get if from generated makefiles. Yes, we can at least provide the build location in an obvious way. We can discuss install locations eventually. OK * I don't generate 'dependencies' (actually the list of files which the buildsystem re-generation depends on) as Aleix did, because there is no well-defined usefulness for that list yet. As Tobias pointed, we at least need to know what files are the part of CMake project, that is, the list of all CMakeLists.txt and *.cmake files, used for generation (ideally, including missing ones, since in that case IDE could be able to tell when missing file is created and refresh the project) As I wrote to Tobias, I'm apprehensive about this, and it would require other work to make cmake parallel safe first. I think if the IDE does not have focus it should not be running 'cmake .' on my behalf. I think if the IDE newly gets focus you can maybe run 'cmake .' at *that* point (after the user is done with their rebase or whatever). That doesn't require giving you a list of files to watch. Maybe I'm missing something though. Agree, the IDE should not do any automatic stuff, when user doesn’t expect it. Though, the use-case here is quite different: Consider a project that have 'add_directory(missing_dir)’ in one of the CMakeLists files. If IDE were aware that missing_dir/CMakeLists.txt is a required file, it might then automatically trigger update, when the user creates this file inside the IDE (not externally). Or it may even have a quick-fix ‘Create missing CMakeList.txt’. Also, files updated and created during VCS (called from the IDE) should be automatically recognized, of course. * Some more information from project() may be relevant, but it's not clear yet. We will likely know more when we have decided the file format and generated some 'interesting' metadata files. Project name, list
Re: [cmake-developers] Generating buildsystem metadata from CMake
On 24 Mar 2015, at 00:54, Stephen Kelly steve...@gmail.com wrote: Tobias Hunger wrote: How about include_directories, compile_definitions and compile_flags? So something along the lines of: include_directories : [/foo, /opt] compile_definitions : [DEF=\Foo\, OTHER_DEF=1] compile_flags: [ -c ] Quoting Anton: Anton Makeev wrote: The other thing that seems troubling to me is that since file, target, language compiler options are split into different parts of metadata, the IDE need to know exactly how to assemble them back into the compiler’s command line (e.g. what flags go first file’s or language’s), duplicating CMake's logic that may be different from version to version and from compiler to compiler. The exact command line is needed to get the actual and precise defines, include search paths etc. from the compiler. Maybe he can chime in with more. I don't really know why the entire command line is needed instead of separate include_directories and compile_definitions arrays as you and I suggested. Perhaps because that would not include -fPIC for example, which causes __PIC__ to be defined. Another item of note is that CMake does not know the compile flags as a sequential container of individual flags currently, but it knows them as a string (that's also why it appears as a string in my generated json currently). The problem with the following format: --- include_directories : [/foo, /opt] compile_definitions : [DEF=\Foo\, OTHER_DEF=1] compile_flags: [ -c ] — Is that it’s incomplete and cannot be used directly: * include directories list misses compiler-defined search paths * quote include directories are not distinguishable * compiler definition do not include compiler-defined(built-in) definitions etc. That is, to have the complete list of include directories and compiler definitions, the IDE will have to call the compiler anyways. And here is where it will need the actual compiler’s command line. Should we had the full list of include directories (split into groups like: , quote, and frameworks) and the list of all compiler defines, we would not need the compiler command-line at all. (It’s actually not completely true, since we’ll need to call the compiler with the correct command-line to get some more info like specific compiler features, but I guess CMake could do it instead of the IDE). Though I don’t this it’s a way to go for CMake generation, since the compiler should be called for every unique command-line flags and undoubtedly make generation way longer. That’s why I think it’s up to the IDE to (lazily) call the compiler when this information is necessary. Does it make sense or maybe I’m missing something? Parsing things is always error prone. Is that -D for definitions or /D? As we can define the format, we could always generate -D and define that as the answer to your question. Anyway, let's see if Anton can explain more about the need for the exact command line and whether my suggestion of duplicating the include_directories and compile_definitions is fine. The compiler flags are definitely needed though. They are used to e.g. decide which dialect of a language are used. This would keep parsing simple and will also provide all the information we need. linker_flags might also be interesting... Yes, something like this is also in the goals for this design. Thanks, Steve. Anton Makeev JetBrains http://www.jetbrains.com Develop with pleasure! smime.p7s Description: S/MIME cryptographic signature -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Generating buildsystem metadata from CMake
On 30 Mar 2015, at 21:21, Alexander Neundorf neund...@kde.org wrote: On Monday, March 30, 2015 19:04:10 Anton Makeev wrote: On 24 Mar 2015, at 00:54, Stephen Kelly steve...@gmail.com wrote: Tobias Hunger wrote: How about include_directories, compile_definitions and compile_flags? So something along the lines of: include_directories : [/foo, /opt] compile_definitions : [DEF=\Foo\, OTHER_DEF=1] compile_flags: [ -c ] Quoting Anton: Anton Makeev wrote: The other thing that seems troubling to me is that since file, target, language compiler options are split into different parts of metadata, the IDE need to know exactly how to assemble them back into the compiler’s command line (e.g. what flags go first file’s or language’s), duplicating CMake's logic that may be different from version to version and from compiler to compiler. The exact command line is needed to get the actual and precise defines, include search paths etc. from the compiler. Maybe he can chime in with more. I don't really know why the entire command line is needed instead of separate include_directories and compile_definitions arrays as you and I suggested. Perhaps because that would not include -fPIC for example, which causes __PIC__ to be defined. Another item of note is that CMake does not know the compile flags as a sequential container of individual flags currently, but it knows them as a string (that's also why it appears as a string in my generated json currently). The problem with the following format: --- include_directories : [/foo, /opt] compile_definitions : [DEF=\Foo\, OTHER_DEF=1] compile_flags: [ -c ] — Is that it’s incomplete and cannot be used directly: * include directories list misses compiler-defined search paths * quote include directories are not distinguishable * compiler definition do not include compiler-defined(built-in) definitions etc. Code for detecting the built-in include dirs and definitions is in CMakeExtraGeneratorDetermineCompilerMacrosAndIncludeDirs.cmake. This is used currently by CMakeFindCodeBlocks.cmake and CMakeFindEclipseCDT4.cmake, i.e. the respective extra-generators, and the information is generated into the eclipse/codeblocks project files. This could be used here too. I don't really understand your second point. Can you elaborate ? Sure, 'some compilers’ distinguish between “include” and include search paths, gcc has -iquote and -I- parameters to specify “include” search path: https://gcc.gnu.org/onlinedocs/cpp/Search-Path.html Distinguishing them is necessary to correctly resolve headers inside the IDE. Additionally, there is frameworks search paths (os OS X), and it’s very important to differentiate them as well. Here is clang’ list just for the reference: --- #include ... search starts here: #include ... search starts here: /Applications/Dev/Xcode-6.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/include /Applications/Dev/Xcode-6.2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include /usr/include /System/Library/Frameworks (framework directory) /Library/Frameworks (framework directory) --- Cheers, Anton Makeev JetBrains http://www.jetbrains.com Develop with pleasure! smime.p7s Description: S/MIME cryptographic signature -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Extracting target metadata, IDE integration
Aleix Pol aleixpol@... writes: I'd really appreciate your feedback on the patch I provided, I'll be happy to provide the information you miss. Aleix Aleix, could you please point me to the most recent patch version? -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
Re: [cmake-developers] Extracting target metadata, IDE integration
Hey everyone, We are developing CLion at JetBrains that utilizes CMake as its main project model. Stephen Kelly timely drew our attention to this discussion, and I’d like to share some thoughts. Here are the general ones and I’ll also comment on separate messages in the thread. We’ve managed to get almost all the necessary project information, using ‘GNU/MinGW Makefiles’ generators: * list of targets comes from 'CMakeFiles/TargetDirectories.txt' * list of source files from ‘CMakeFiles/A.dir/DependInfo.cmake’ * list of headers and resources from ‘codeblocks.cbp’ file * per-file/target compiler and its flags from ‘CMakeFiles/Target.dir/flags.make’ * location of product files from ‘CMakeFiles/Target.dir/build.make’ What we miss (not all of them are directly related the targets metadata, but I'd like to share anyway since the topic is about IDE integration): * Location of every source file and target in CMakeLists - it would greatly help to refactorings, navigation etc. * Complete list of headers and resource files - they are only listed in special generators like code blocks. * An easily parseable list of errors and warnings with origin information. At the moment we parse error output but not everything can be parsed reliably. * No progress indication. Since the generation may take several minutes, providing feedback is crucial. * Ability to distinguish a library from an executable target. This will help to offer a better UI for run/debug configurations. * Possibility to collect information for every build target in one run. Currently, we have to invoke generator for every CMAKE_BUILT_TYPE separately. * CMake stops processing when it find a missing file, so that IDE cannot have full project model, until this files is created. * When there are existing in-source generated files in the project dir, CMake doesn't allow generating into a separate out-of-source folder. An IDE has to invent workarounds here. * Not sure if it’s possible at all - a lightweight phase where CMake only collects necessary information (list of files/targets, compiler settings). This will help IDE react to the changes much faster. Here is why I think the discussed functionality should not be a separate generator: CLion doesn’t have it’s own project model nor is intended as build tool replacement. Currently, the only option for CLion users is makefiles build, that is not a best option for everyone: there is a good and fast alternative ‘Ninja’. Ideally, users should be able to choose whatever tool better suites them. The problem is that the generated Makefiles are used both, for internal needs, like reading project structure, and also to build and run the user’s program. If we wanted to support Ninja generator, we would need to rewrite all the logic that retrieves the project information, using the files that are created by Ninja generator. While I suppose it’s possible, it’s not the best option - very error prone and resource demanding. If CMake generated a separate descriptor, regardless the generator used (Makefiles, Ninja), it would be much easier to support; and the users will benefit from better, more reliable and faster CMake integration in an IDE. Regards, Anton Makeev -- Powered by www.kitware.com Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ Kitware offers various services to support the CMake community. For more information on each offering, please visit: CMake Support: http://cmake.org/cmake/help/support.html CMake Consulting: http://cmake.org/cmake/help/consulting.html CMake Training Courses: http://cmake.org/cmake/help/training.html Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Follow this link to subscribe/unsubscribe: http://public.kitware.com/mailman/listinfo/cmake-developers
