Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-08-30 Thread David Blaikie 
On Fri, Aug 24, 2018 at 2:35 AM Stephen Kelly  wrote:

>
> On 24/08/18 02:32, David Blaikie wrote:
>
> On Tue, Jul 24, 2018 at 3:20 PM Stephen Kelly  wrote:
>
>> David Blaikie wrote:
>>
>> > (just CC'ing you Richard in case you want to read my ramblings/spot any
>> > inaccuracies, etc)
>> >
>> > Excuse the delay - coming back to this a bit now. Though the varying
>> > opinions on what modules will take to integrate with build system still
>> > weighs on me a bit
>>
>> Can you explain what you mean by 'weighs on you'? Does that mean you see
>> it
>> as tricky now?
>
>
> Yes, to some extent. If the build system is going to require the
> compiler-callsback-to-buildsystem that it sounds like (from discussions
> with Richard & Nathan, etc) is reasonable - yeah, I'd say that's a bigger
> change to the way C++ is compiled than I was expecting/thinking of going
> into this.
>
>
> Yes.
>
>
>
>
>> I've kind of been assuming that people generally think it is not tricky,
>> and
>> I'm just wrong in thinking it is and I'll eventually see how it is all
>> manageable.
>>
>
> I think it's manageable - the thing that weighs on me, I suppose, is
> whether or not the community at large will "buy" it, as such.
>
>
> Yes, that has been my point since I first started talking about modules. I
> don't think modules will gain a critical mass of adoption as currently
> designed (and as currently designed to work with buildsystems).
>

For myself, I don't think I'd go that far (I think the current design might
be feasible) - and I'm mostly trying to set aside those concerns to get to
more concrete things - to work through the build system ramifications,
prototype things, etc, to get more concrete experience/demonstration of the
possibilities and problems.

> And part of that is on the work we're doing to figure out the integration
> with build systems, etc, so that there's at least the first few pieces of
> support that might help gain user adoption to justify/encourage/provide
> work on further support, etc...
>
> Yes, reading the document Nathan sent us on June 12th this year, it seems
> that CMake would have to implement a server mode so that the compiler will
> invoke it with RPC. That server will also need to consume some data
> generated by CMake during buildsystem generation (eg user specified flags)
> and put that together with information sent by the compiler (eg ) in order
> to formulate a response. It's complex. Maybe CMake and other buildsystem
> generators can do it, but there are many bespoke systems out there which
> would have to have some way to justify the cost of developing such a thing.
>

Yeah, certainly a possibility - maybe it'd be enough of a wedge to cause
people to collapse the large build system space into fewer options - many
other languages require this sort of level of coupling between
compiler/build system/language.

But getting that momentum started would be getting the main build systems
supporting it & starting at the leaves (independent projects starting to
write modular code for themselves - even if all their external dependencies
aren't) - then, with enough users using it downstream, might be some
libraries providing a modular option (or maybe even only provide it as
modules & that'd be the wedge I'm talking about - then teams/projects that
don't support modules would be left out a bit & provide an incentive for
them to move over to have modules support to use these dependencies).

> > The build.sh script shows the commands required to build it (though I
>> > haven't checked the exact fmodule-file dependencies to check that
>> they're
>> > all necessary, etc) - and with current Clang top-of-tree it does build
>> and
>> > run the example dinnerparty program.
>>
>> Ok. I tried with my several-weeks-old checkout and it failed on the first
>> command with -modules-ts in it (for AbstractFruit.cppm - the simplest
>> one).
>>
>> I'll update my build and try again, but that will take some time.
>>
>
> Huh - I mean it's certainly a moving target - I had to file/workaround a
> few bugs to get it working as much as it is, so not /too/ surprising. Did
> you get it working in the end? If not, could you specify the exact revision
> your compiler's at and show the complete output?
>
>
> Yes, I got it working. See
>
>  https://www.mail-archive.com/cmake-developers@cmake.org/msg18623.html
>
>
>
> > But I'm not sure how best to determine the order in which to build files
>> within a library - that's where the sort of -MM-esque stuff, etc, would be
>
>
>> > necessary.
>>
>> Would it? I thought the -MM stuff would mostly be necessary for
>> determining
>> when to rebuild? Don't we need to determine the build order before the
>> first
>> build of anything? The -MM stuff doesn't help that.
>>
>
> -MM produces output separate from the compilation (so far as I can tell -
> clang++ -MM x.cpp doesn't produce anything other than the makefile fragment
> on stdout) & finds all the headers, etc. So that's basically the same

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-08-24 Thread Stephen Kelly 


On 24/08/18 02:32, David Blaikie wrote:
On Tue, Jul 24, 2018 at 3:20 PM Stephen Kelly > wrote:


David Blaikie wrote:

> (just CC'ing you Richard in case you want to read my
ramblings/spot any
> inaccuracies, etc)
>
> Excuse the delay - coming back to this a bit now. Though the varying
> opinions on what modules will take to integrate with build
system still
> weighs on me a bit

Can you explain what you mean by 'weighs on you'? Does that mean
you see it
as tricky now?


Yes, to some extent. If the build system is going to require the 
compiler-callsback-to-buildsystem that it sounds like (from 
discussions with Richard & Nathan, etc) is reasonable - yeah, I'd say 
that's a bigger change to the way C++ is compiled than I was 
expecting/thinking of going into this.


Yes.


I've kind of been assuming that people generally think it is not
tricky, and
I'm just wrong in thinking it is and I'll eventually see how it is
all
manageable.


I think it's manageable - the thing that weighs on me, I suppose, is 
whether or not the community at large will "buy" it, as such.


Yes, that has been my point since I first started talking about modules. 
I don't think modules will gain a critical mass of adoption as currently 
designed (and as currently designed to work with buildsystems).


And part of that is on the work we're doing to figure out the 
integration with build systems, etc, so that there's at least the 
first few pieces of support that might help gain user adoption to 
justify/encourage/provide work on further support, etc...


Yes, reading the document Nathan sent us on June 12th this year, it 
seems that CMake would have to implement a server mode so that the 
compiler will invoke it with RPC. That server will also need to consume 
some data generated by CMake during buildsystem generation (eg user 
specified flags) and put that together with information sent by the 
compiler (eg ) in order to formulate a response. It's complex. Maybe 
CMake and other buildsystem generators can do it, but there are many 
bespoke systems out there which would have to have some way to justify 
the cost of developing such a thing.



> The build.sh script shows the commands required to build it
(though I
> haven't checked the exact fmodule-file dependencies to check
that they're
> all necessary, etc) - and with current Clang top-of-tree it does
build and
> run the example dinnerparty program.

Ok. I tried with my several-weeks-old checkout and it failed on
the first
command with -modules-ts in it (for AbstractFruit.cppm - the
simplest one).

I'll update my build and try again, but that will take some time.


Huh - I mean it's certainly a moving target - I had to file/workaround 
a few bugs to get it working as much as it is, so not /too/ 
surprising. Did you get it working in the end? If not, could you 
specify the exact revision your compiler's at and show the complete 
output?


Yes, I got it working. See

 https://www.mail-archive.com/cmake-developers@cmake.org/msg18623.html



> But I'm not sure how best to determine the order in which to
build files within a library - that's where the sort of -MM-esque
stuff, etc, would be
> necessary.

Would it? I thought the -MM stuff would mostly be necessary for
determining
when to rebuild? Don't we need to determine the build order before
the first
build of anything? The -MM stuff doesn't help that.


-MM produces output separate from the compilation (so far as I can 
tell - clang++ -MM x.cpp doesn't produce anything other than the 
makefile fragment on stdout) & finds all the headers, etc. So that's 
basically the same as what we'd need here


Are you sure? I thought compiling with -MM gives us information that we 
need before we compile the first time. Sorry if that was not clear from 
what I wrote above. I see a chicken-egg problem. However, I assume I'm 
just misunderstanding you (you said that -MM would be used to determine 
build order for the initial build) so let's just drop this.


Looking at your example - if you have a library for all the fruits and 
libabstractfruit, libfruitsalad, libnotfruitsalad, and libbowls - then 
you'd have one module interface for each of those (AbstractFruit.cppm, 
FruitSalad.cppm, NotFruitSalad.cppm, Bowls.cppm) that would be 
imported (so replace "import Apple", "import Grape" with "import 
FruitSalad", etc... ) & the implementations could be in multiple files 
if desired (Apple.cpp, Grape.cpp, etc).


Could you show me what that would look like for the repo? I am 
interested to know if this approach means concatenating the content of 
multiple files (eg Grape.h and Apple.h) and porting that result to a 
module. My instinct says that won't gain adoption.



>> Ok. That's not much better though. It still means
editing/generating the
>> buildsystem each

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-08-23 Thread David Blaikie 
On Tue, Jul 24, 2018 at 3:20 PM Stephen Kelly  wrote:

> David Blaikie wrote:
>
> > (just CC'ing you Richard in case you want to read my ramblings/spot any
> > inaccuracies, etc)
> >
> > Excuse the delay - coming back to this a bit now. Though the varying
> > opinions on what modules will take to integrate with build system still
> > weighs on me a bit
>
> Can you explain what you mean by 'weighs on you'? Does that mean you see
> it
> as tricky now?


Yes, to some extent. If the build system is going to require the
compiler-callsback-to-buildsystem that it sounds like (from discussions
with Richard & Nathan, etc) is reasonable - yeah, I'd say that's a bigger
change to the way C++ is compiled than I was expecting/thinking of going
into this.

Some folks see this as not "tricky" but just "hey, C++ has been getting
away with a vastly simplified build model until now - now that it's getting
modular separation like other languages, it's going to have to have a build
system like other languages" (ala Java/C#, Haskell/Go maybe, etc) - but I'm
not especially familiar with any other languages enterprise-level build
systems (done a bit of C# at Microsoft but wasn't looking closely at the
build system - insulated from it by either Visual Studio or the OS release
processes, etc - and the last time I did Java I was working on small enough
things & without many cores, so throwing all the .java files in a package
at the compiler in one go was totally reasonable)


> I've kind of been assuming that people generally think it is not tricky,
> and
> I'm just wrong in thinking it is and I'll eventually see how it is all
> manageable.
>

I think it's manageable - the thing that weighs on me, I suppose, is
whether or not the community at large will "buy" it, as such. And part of
that is on the work we're doing to figure out the integration with build
systems, etc, so that there's at least the first few pieces of support that
might help gain user adoption to justify/encourage/provide work on further
support, etc...

> To that end, Stephen, I've made a fork of your example repository & a very
> > simple/direct change to use C++ modules as currently implemented in
> Clang.
>
> Interesting, thanks for doing that! Here's the link for anyone else
> interested:
>
>  https://github.com/dwblaikie/ModulesExperiments


Oh, thanks - totally didn't realize I failed to link that!


> > The build.sh script shows the commands required to build it (though I
> > haven't checked the exact fmodule-file dependencies to check that they're
> > all necessary, etc) - and with current Clang top-of-tree it does build
> and
> > run the example dinnerparty program.
>
> Ok. I tried with my several-weeks-old checkout and it failed on the first
> command with -modules-ts in it (for AbstractFruit.cppm - the simplest one).
>
> I'll update my build and try again, but that will take some time.
>

Huh - I mean it's certainly a moving target - I had to file/workaround a
few bugs to get it working as much as it is, so not /too/ surprising. Did
you get it working in the end? If not, could you specify the exact revision
your compiler's at and show the complete output?


> > If you happen to try experimenting with any ways the commands in the
> > build.sh file could be run from CMake it a sensible way - even if you
> > hypothesize what -MM support (or other compiler hooks like the dependency
> > server I alluded to above, etc) for modules might look like to do so, I'd
> > love to chat about it/throw ideas around/try mocking up/prototyping the
> > sort of compiler support (I don't think there's -MM support yet, but I
> > could see about adding it, for example) that seems like it might be most
> > useful.
>
> Yes, that would be useful I think.
>
> > One thing I'm vaguely concerned about is actually the dependency of
> > building modules within a single library (as in this project/example - at
> > least the way I've built it for now - I didn't try building it as
> separate
> > .so/.a files). At least across-library we can work at the library
> > granularity and provide on the command line (or via a file as GCC does)
> > the module files for all the modules from dependent libraries. But I'm
> not
> > sure how best to determine the order in which to build files within a
> > library
>
> Exactly, that's one of the reasons my repo has libraries of multiple files
> with dependencies between them, but building everything into one
> executable
> also exposes that issue.
>

Ah, when I say "library" I mean static (.lib/.a) or dynamic (.so/.dll/etc)
- same situation in either. But yeah - any situation where there are
multiple modules within a single library, getting those dependencies will
involve some discovery (either a pre-parsing check (same way a build system
might check which headers are included in a file) or the compiler callback
system, where the compiler asks the build system for the modules it needs)
- and even in the one-module-per-library


> I guess it would work in a

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-08-07 Thread David Blaikie 
On Mon, Aug 6, 2018 at 2:37 PM Stephen Kelly  wrote:

> Stephen Kelly wrote:
> >> The build.sh script shows the commands required to build it (though I
> >> haven't checked the exact fmodule-file dependencies to check that
> they're
> >> all necessary, etc) - and with current Clang top-of-tree it does build
> >> and run the example dinnerparty program.
> >
> > Ok. I tried with my several-weeks-old checkout and it failed on the first
> > command with -modules-ts in it (for AbstractFruit.cppm - the simplest
> > one).
> >
> > I'll update my build and try again, but that will take some time.
>
> I have locally tried your modifications. Aside from the content of my
> previous email, I updated my clone (force push) to clean up the commits,
> and
> to modify your build.sh script to maintain the different libraries in the
> repo.
>
>  https://github.com/steveire/ModulesExperiments/commits/master
>
> I am still interested in what the C++ code (and build.sh) look like in the
> case of one-module-per-library.
>

Oh, sure - that's easy enough, in terms of rolling together any modules in
the same library. Mash all the code in any cppm interface files that are
part of the same library into a single cppm file & build that into a single
module.


> It is obvious from looking at build.sh as it is that the buildsystem needs
> to be changed when adding a new import to a c++ file, as I have described.
>

Though potentially not manually - and some of those lines would be added
when the new module was added (as would be done when a new .cpp file is
added today) - rather than when the import occurred.

The 'worst' part is for intra-library module dependencies. For
inter-library module dependencies, a coarse-grained solution would be to
only build a library once all of the BMIs (.pcm files) for dependent
libraries are built - and all of those are passed to libraries that depend
on them. But, yeah, having the build system trying to figure out the
dependency between modules within a library (which isn't explicit in the
build files, unlike the inter-module dependency) is awkward. (& honestly,
for maximum build parallelism, you wouldn't want the coarse-grained
solution I described above - because then if something in library A
depended on only module B.1 but not module B.2, you'd want to start
building that part of A when B.1's BMI was built, without waiting for B.2's
BMI to be finished)

So, yes, if you have more than one module per library, then both for the
intra-library module dependency problem and the minimal-cross-library
dependency issue, you'd need the build system to either be told by the
compiler as its going (ala the proposed build server/oracle system that's
implemented in GCC) or by pre-scanning with a semi-aware scanner (doesn't
have to know all of C++ - modules are notionally well defined to be able to
scan a short preamble with limited syntax... but I'm not fully up on all
the nuance there & retro #include-to-import legacy support might complicate
things)

(I guess, technically, even in the one-module-per-library, arguably your
library dependency courld be a purely implementation dependency, in which
case you'd still want to be able to build your 'dependent' BMI without
waiting for the dependency's BMI to finish, since they don't depend on each
other - so if your build system doesn't differentiate between external and
internal dependencies, then again you'd need the kind of discovery phase if
you want maximal build parallelism... )


> see the commit adding Grape separately and the changes required to the
> buildsystem which were not required in the non-modules world:
>
>  https://github.com/steveire/ModulesExperiments/commit/428bea53fc6
>
> I will see if I can get a recent GCC build for comparison and to determine
> whether the callback-to-the-buildsystem used in GCC makes a difference in
> that respect.
>
> I'm still interested in a response to my previous email in that respect.
>

Yep, coming back to that now.


>
> Thanks,
>
> Stephen.
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake-developers
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting:

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-08-06 Thread Stephen Kelly 
Stephen Kelly wrote:
>> The build.sh script shows the commands required to build it (though I
>> haven't checked the exact fmodule-file dependencies to check that they're
>> all necessary, etc) - and with current Clang top-of-tree it does build
>> and run the example dinnerparty program.
> 
> Ok. I tried with my several-weeks-old checkout and it failed on the first
> command with -modules-ts in it (for AbstractFruit.cppm - the simplest
> one).
> 
> I'll update my build and try again, but that will take some time.

I have locally tried your modifications. Aside from the content of my 
previous email, I updated my clone (force push) to clean up the commits, and 
to modify your build.sh script to maintain the different libraries in the 
repo.

 https://github.com/steveire/ModulesExperiments/commits/master

I am still interested in what the C++ code (and build.sh) look like in the 
case of one-module-per-library. 

It is obvious from looking at build.sh as it is that the buildsystem needs 
to be changed when adding a new import to a c++ file, as I have described. 
see the commit adding Grape separately and the changes required to the 
buildsystem which were not required in the non-modules world:

 https://github.com/steveire/ModulesExperiments/commit/428bea53fc6

I will see if I can get a recent GCC build for comparison and to determine 
whether the callback-to-the-buildsystem used in GCC makes a difference in 
that respect.

I'm still interested in a response to my previous email in that respect.

Thanks,

Stephen.


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-07-24 Thread Stephen Kelly 
David Blaikie wrote:

> (just CC'ing you Richard in case you want to read my ramblings/spot any
> inaccuracies, etc)
> 
> Excuse the delay - coming back to this a bit now. Though the varying
> opinions on what modules will take to integrate with build system still
> weighs on me a bit

Can you explain what you mean by 'weighs on you'? Does that mean you see it 
as tricky now? 

I've kind of been assuming that people generally think it is not tricky, and 
I'm just wrong in thinking it is and I'll eventually see how it is all 
manageable.

> - but I'm trying to find small ways/concrete steps to
> make some progress on this rather than being lost in choice/opinion
> paralysis.

Cool.

> To that end, Stephen, I've made a fork of your example repository & a very
> simple/direct change to use C++ modules as currently implemented in Clang.

Interesting, thanks for doing that! Here's the link for anyone else 
interested:

 https://github.com/dwblaikie/ModulesExperiments

> The build.sh script shows the commands required to build it (though I
> haven't checked the exact fmodule-file dependencies to check that they're
> all necessary, etc) - and with current Clang top-of-tree it does build and
> run the example dinnerparty program.

Ok. I tried with my several-weeks-old checkout and it failed on the first 
command with -modules-ts in it (for AbstractFruit.cppm - the simplest one).

I'll update my build and try again, but that will take some time.

> If you happen to try experimenting with any ways the commands in the
> build.sh file could be run from CMake it a sensible way - even if you
> hypothesize what -MM support (or other compiler hooks like the dependency
> server I alluded to above, etc) for modules might look like to do so, I'd
> love to chat about it/throw ideas around/try mocking up/prototyping the
> sort of compiler support (I don't think there's -MM support yet, but I
> could see about adding it, for example) that seems like it might be most
> useful.

Yes, that would be useful I think.

> One thing I'm vaguely concerned about is actually the dependency of
> building modules within a single library (as in this project/example - at
> least the way I've built it for now - I didn't try building it as separate
> .so/.a files). At least across-library we can work at the library
> granularity and provide on the command line (or via a file as GCC does)
> the module files for all the modules from dependent libraries. But I'm not
> sure how best to determine the order in which to build files within a
> library

Exactly, that's one of the reasons my repo has libraries of multiple files 
with dependencies between them, but building everything into one executable 
also exposes that issue.

I guess it would work in a similar way to determining the order to link 
libraries though, so probably not a major problem.

> - that's where the sort of -MM-esque stuff, etc, would be
> necessary.

Would it? I thought the -MM stuff would mostly be necessary for determining 
when to rebuild? Don't we need to determine the build order before the first 
build of anything? The -MM stuff doesn't help that.

>> It seems quite common to have one PCH file per shared library (that's
>> what Qt does for example). What makes you so sure that won't be the case
>> with modules?
>>
> 
> Can't say I've worked with code using existing PCH - if that seems common
> enough, it might be a good analogy/guidance people might follow with C++
> modules.

Perhaps. I wonder how the C++ code/build.sh code would change in that 
scenario? If there were only four modules - one for each of the libraries 
(as delimited in the CMakeLists). Would the C++ code change then too 
(something about building partial modules from each of the multiple cppm 
files?), or how would the build.sh code change?

>> Ok. That's not much better though. It still means editing/generating the
>> buildsystem each time you add an import.
> 
> 
> Isn't that true today with headers, though? 

No. Imagine you implemented FruitBowl.cpp in revision 1 such that it did not 
#include Grape.h and it did not add the Grape to the bowl.

Then you edit FruitBowl.cpp to #include Grape.h and add the Grape to the 
bowl. Because Grape.h and Apple.h are in the same directory (which you 
already have a -Ipath/to/headers for in your buildsystem), in this (today) 
scenario, you don't have to edit the buildsystem.

In your port, you would have to add an import of Grape (fine, equivalent), 
add the Grape to the bowl (the same as today), but additionally, you have to 

 * add -fmodule-file=Grape.pcm to the compile line or run your buildsystem 
generator such as CMake to cause that compile line to be generated with the 
argument added.
 * Generate Grape.pcm (because the library has 1000 fruit classes in it and 
your buildsystem is smart enough to lazily generate pcm files as needed)

Partly that is a result of the totally-granular approach you took to 
creating modules (one module per class). If you used the

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-07-19 Thread David Blaikie 
(just CC'ing you Richard in case you want to read my ramblings/spot any
inaccuracies, etc)

Excuse the delay - coming back to this a bit now. Though the varying
opinions on what modules will take to integrate with build system still
weighs on me a bit - but I'm trying to find small ways/concrete steps to
make some progress on this rather than being lost in choice/opinion
paralysis.

To that end, Stephen, I've made a fork of your example repository & a very
simple/direct change to use C++ modules as currently implemented in Clang.
Some workarounds are required for a few bugs/incomplete features (oh, one I
didn't comment in source is the use of #include for the standard library -
that'd actually be "import legacy" in the current modules TS2/atom merged
proposal, if I understand correctly - and the way I've implemented it in
the example is as-if the standard library were not modularized (so the
project wraps it in a modularized header itself))

The build.sh script shows the commands required to build it (though I
haven't checked the exact fmodule-file dependencies to check that they're
all necessary, etc) - and with current Clang top-of-tree it does build and
run the example dinnerparty program.

There are a few ideas being tossed aroudn currently for how module
dependency discovery could be done by build systems or exposed by the
compiler itself (GCC has a service protocol it can interact with when it
needs a new compiled module description, which the build system could
implement to fulfill such requests - giving the build system all
information about the inter-module dependencies without a separate ahead of
time scan and while allowing maximal parallelism (compiler could request
all needed modules then wait for them all to be ready - rather than one at
a time, so as not to stall parallelism)).

The syntax is intended to support a build system that would do the scanning
itself, though - requiring limited preprocessing (I think the preprocessing
would need to know where to stop - since it might not be able to preprocess
the whole file without error without importing modules (since those imports
could contain new macro definitions used later in the file)).

If you happen to try experimenting with any ways the commands in the
build.sh file could be run from CMake it a sensible way - even if you
hypothesize what -MM support (or other compiler hooks like the dependency
server I alluded to above, etc) for modules might look like to do so, I'd
love to chat about it/throw ideas around/try mocking up/prototyping the
sort of compiler support (I don't think there's -MM support yet, but I
could see about adding it, for example) that seems like it might be most
useful.

One thing I'm vaguely concerned about is actually the dependency of
building modules within a single library (as in this project/example - at
least the way I've built it for now - I didn't try building it as separate
.so/.a files). At least across-library we can work at the library
granularity and provide on the command line (or via a file as GCC does) the
module files for all the modules from dependent libraries. But I'm not sure
how best to determine the order in which to build files within a library -
that's where the sort of -MM-esque stuff, etc, would be necessary. (though
that sort of stuff would be useful even cross-library to speed up the build
(eg: foo_a.cppm depends on bar_a.cppm but not bar_b.cppm - don't rebuild
foo_a.cppm if bar_b.cppm changes, even though libfoo depends on libbar, etc)

On Tue, May 15, 2018 at 1:34 AM Stephen Kelly  wrote:

> David Blaikie wrote:
>
> >> Nope, scratch that ^ I had thought that was the case, but talking more
> >> with Richard Smith it seems there's an expectation that modules will be
> >> somewhere between header and library granularity (obviously some small
> >> libraries today have one or only a few headers, some (like Qt) have many
> >> - maybe those on the Qt end might have slightly fewer modules than the
> >> have headers - but still several modules to one library most likely, by
> >> the sounds of it)
> >>
> >>
> >> Why? Richard maybe you can answer that? These are the kinds of things I
> >> was trying to get answers to in the previous post to iso sg2 in the
> >> google group. I didn't get an answer as definitive as this, so maybe you
> >> can share the reason behind such a definitive answer?
> >>
> >
> > It's more that the functionality will allow this & just judging by how
> > people do things today (existing header granularity partly motivated by
> > the cost of headers that doesn't apply to modules), how they're likely to
> > do things in the future (I personally would guess people will probably
> try
> > to just port their headers to modules - and a few places where there are
> > circular dependencies in headers or the like they might glob them up into
> > one module).
>
>
> It seems quite common to have one PCH file per shared library (that's what
> Qt does for example). What makes you so sure

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-05-15 Thread Brad King 
On 05/15/2018 03:22 AM, Stephen Kelly wrote:
> So, the answer for cmake might be that CMake can learn to extract that 
> stuff, but ignore certain cases like imports within ifdefs. 

We'd need to do the extraction from already-preprocessed sources.
This is how Fortran+Ninja+CMake works.  Unfortunately for C++
this will typically require preprocessing twice: once just to
extract module dependencies and again to actually compile.  With
Fortran we compile using the already-preprocessed source but
doing that with C++ will break things like Clang's nice handling
of macros in diagnostic messages.

-Brad
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-05-15 Thread Stephen Kelly 
David Blaikie wrote:

>> Nope, scratch that ^ I had thought that was the case, but talking more
>> with Richard Smith it seems there's an expectation that modules will be
>> somewhere between header and library granularity (obviously some small
>> libraries today have one or only a few headers, some (like Qt) have many
>> - maybe those on the Qt end might have slightly fewer modules than the
>> have headers - but still several modules to one library most likely, by
>> the sounds of it)
>>
>>
>> Why? Richard maybe you can answer that? These are the kinds of things I
>> was trying to get answers to in the previous post to iso sg2 in the
>> google group. I didn't get an answer as definitive as this, so maybe you
>> can share the reason behind such a definitive answer?
>>
> 
> It's more that the functionality will allow this & just judging by how
> people do things today (existing header granularity partly motivated by
> the cost of headers that doesn't apply to modules), how they're likely to
> do things in the future (I personally would guess people will probably try
> to just port their headers to modules - and a few places where there are
> circular dependencies in headers or the like they might glob them up into
> one module).


It seems quite common to have one PCH file per shared library (that's what 
Qt does for example). What makes you so sure that won't be the case with 
modules?

I'd say that what people will do will be determined by whatever their tools 
optimize for. If it is necessary to list all used modules on the compile 
line, people would choose fewer modules. If 'import QtCore' is fast and 
allows the use of QString and QVariant etc and there is no downside, then 
that will be the granularity offered by Qt (instead of 'QtCore.QString'). 
That is also comparable to '#include ' which is possible today.

>> I just looked through the commits from Boris, and it seems he made some
>> changes relating to -fmodule-file=. That still presupposes that all
>> (transitively) used module files are specified on the command line.
>>
> 
> Actually I believe the need is only the immediate dependencies - at least
> with Clang's implementation.

Ok. That's not much better though. It still means editing/generating the 
buildsystem each time you add an import. I don't think a model with that 
requirement will gain adoption.

>> I was talking about the -fprebuilt-module-path option added by Manman Ren
>> in https://reviews.llvm.org/D23125 because that actually relieves the
>> user/buildsystem of maintaining a list of all used modules (I hope).
>>
> 
> *nod* & as you say, GCC has something similar. Though the build system
> probably wants to know about the used modules to do dependency analysis &
> rebuilding correctly. 

Yes, presumably that will work with -MM.

> Yeah, thanks for the link - useful to read.

There seems to be a slew of activity around modules at the moment. You can 
read some other reactions here which might have input for your paper:

https://www.reddit.com/r/cpp/comments/8jb0nt/what_modules_can_actually_provide_and_what_not/

https://www.reddit.com/r/cpp/comments/8j1edf/really_think_that_the_macro_story_in_modules_is/

I look forward to reading your paper anyway.

>> I think clang outputs the definitions in a separate object file, but GCC
>> currently doesn't. Perhaps that's a difference that cmake has to account
>> for or pass on to the user.
>>
> 
> Clang outputs frontend-usable (not object code, but serialized AST usable
> for compiling other source code) descriptions of the entire module
> (whatever it contains - declarations, definitions, etc) to the .pcm file.
> It can then, in a separate step, build an object file from the pcm. I
> think GCC produces both of these artifacts in one go - but not in the same
> file.

Ok, I must have misremembered something.


>> Sure. I didn't notice anything from reading, but I also didn't try it
>> out. You might need to provide a repo with the module.modulemap/c++ files
>> etc that are part of your experiment. Or better, provide something based
>> on modules-ts that I can try out.
>>
> 
> *nod* I'll see if I can get enough of modules-ts type things working to
> provide some examples, but there's some more variance/uncertainty there in
> the compiler support, etc.

Something working only with clang for example would be a good start.

>> I'm guessing that's enough for you to implement what you want as an
>> experiment?
>>
> 
> OK, so in that case it requires source changes to cmake? *nod* sounds
> plausible - I appreciate the pointers. I take it that implies there's not
> a way I could hook into those file kinds and filters without changing
> cmake? (ie: from within my project's cmake build files, without modifying
> a cmake release)

There is no way to hook into the system I described without patching CMake. 
Your custom command approach might be the way to do that if it is the 
priority.

Thanks,

Stephen.


-- 

Powered by www.kitware.com

Please keep

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-05-15 Thread Stephen Kelly 
Brad King wrote:

> On 05/07/2018 12:01 PM, Stephen Kelly wrote:
>> Hopefully Brad or someone else can provide other input from research
>> already done.
> 
> I'm not particularly familiar with what compiler writers or the modules
> standard specification expects build systems to do w.r.t modules.
> However, IIUC at least at one time the expectation was that the module
> files would not be installed like headers and are used only within a
> local build tree.  Are modules even supposed to be first-class entities
> in the build system specification that users write?

The answer is probably both 'hopefully not' and 'sometimes'.

> In the Fortran world users just list all the sources and build systems are
> expected to figure it out.  CMake has very good support for Fortran
> modules. Our Ninja generator has rules to preprocess the translation units
> first, then parse the preprocessed output to extract module definitions
> and usages, then inject the added dependencies into the build graph, and
> then begin compilation of sources ordered by those dependencies (this
> requires a custom fork of Ninja pending upstream acceptance).
> 
> Is that what is expected from C++ buildsystems for modules too?


Hopefully. However, in some cases, the step of 'extracting module 
definitions and usages' might be very hard to do. This document is quite 
concise about that:

 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1052r0.html

So, the answer for cmake might be that CMake can learn to extract that 
stuff, but ignore certain cases like imports within ifdefs. Maybe CMake 
could then also provide API for users to specify the usages/dependencies 
explicitly in those cases. I don't know how convenient that would be (or 
could be made through design).

Thanks,

Stephen.


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-05-09 Thread David Blaikie 
On Mon, May 7, 2018 at 10:13 AM Brad King  wrote:

> On 05/07/2018 12:01 PM, Stephen Kelly wrote:
> > Hopefully Brad or someone else can provide other input from research
> already done.
>
> I'm not particularly familiar with what compiler writers or the modules
> standard specification expects build systems to do w.r.t modules.
> However, IIUC at least at one time the expectation was that the module
> files would not be installed like headers


The module interface source file is, to the best of my knowledge, intended
to be installed like headers - and I'm currently advocating/leaning/pushing
towards it being installed exactly that way (in the same directories, using
the same include search path, etc).


> and are used only within a local build tree.


The /binary/ representation of the module interface is likely to be only
local to a specific build tree - since it's not portable between compilers
or different compiler flag sets, even.


>   Are modules even supposed to be first-class entities
> in the build system specification that users write?
>
> In the Fortran world users just list all the sources and build systems are
> expected to figure it out.  CMake has very good support for Fortran
> modules.
> Our Ninja generator has rules to preprocess the translation units first,
> then parse the preprocessed output to extract module definitions and
> usages,
> then inject the added dependencies into the build graph, and then begin
> compilation of sources ordered by those dependencies (this requires a
> custom fork of Ninja pending upstream acceptance).
>
> Is that what is expected from C++ buildsystems for modules too?
>

Yes, likely something along those lines - though I'm looking at a few
different possible support models. A couple of major different ones (that
may be both supported by GCC and Clang at least, if they work out/make
sense) are:

* the wrapper-script approach, where, once the compiler determines the set
of direct module dependencies, it would invoke a script to ask for the
location of the binary module interface files for those modules. Build
systems could use this to dynamically discover the module dependencies of a
file as it is being compiled.

* tool-based parsing (more like what you've described Fortran+Ninja+CMake
is doing). The goal is to limit the syntax of modules code enough that
discovering the set of direct module dependencies is practical for an
external (non-compiler) tool - much like just some preprocessing and
looking for relatively simple keywords, etc. - then the tool/build system
can find the dependencies ahead of time without running the compiler

(a 3rd scenario, is what I've been sort of calling the "hello world"
example - where it's probably important that it's still practical for a new
user to compile something simple like "hello world" that uses a modularized
standard library, without having to use a build system for it (ie: just run
the compiler & it goes off & builds the in-memory equivalent of BMIs
without even writing them to disk/reusing them in any way))

- Dave


>
> -Brad
>
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-05-09 Thread David Blaikie 
Forwarding to the developers list, since my original reply bounced as I
wasn't subscribed to the list yet.

-- Forwarded message -
From: David Blaikie 
Date: Wed, May 9, 2018 at 3:23 PM
Subject: Re: Experiments in CMake support for Clang (header & standard)
modules
To: Stephen Kelly 
Cc: cmake-developers@cmake.org , Richard Smith <
rich...@metafoo.co.uk>, 


On Mon, May 7, 2018 at 9:01 AM Stephen Kelly  wrote:

>
> I think this discussion is more suited to the cmake-developers mailing
> list. Moving there.
>

Sure thing - I was sort of hoping much/all of this functionality could be
implemented at the user-level without changes to cmake itself at least
initially - then upstream convenience mechanisms so everyone doesn't have
to rewrite the functionality locally.


> Hopefully Brad or someone else can provide other input from research
> already done.
>
>
> On 05/07/2018 12:49 AM, David Blaikie wrote:
>
>
> The basic commands required are:
>>
>>   clang++ -fmodules -xc++ -Xclang -emit-module -Xclang -fmodules-codegen
>> -fmodule-name=foo foo.modulemap -o foo.pcm
>>   clang++ -fmodules -c -fmodule-file=foo.pcm use.cpp
>>   clang++ -c foo.pcm
>>   clang++ foo.o use.o -o a.out
>>
>>
>> Ok. Fundamentally, I am suspicious of having to have a
>> -fmodule-file=foo.pcm for every 'import foo' in each cpp file. I shouldn't
>> have to manually add that each time I add a new import to my cpp file. Even
>> if it can be automated (eg by CMake), I shouldn't have to have my
>> buildsystem be regenerated each time I add an import to my cpp file either.
>>
>> That's something I mentioned in the google groups post I made which you
>> linked to. How will that work when using Qt or any other library?
>>
>
> - My understanding/feeling is that this would be similar to how a user has
> to change their link command when they pick up a new dependency.
>
>
> Perhaps it would be interesting to get an idea of how often users need to
> change their buildsystems because of a new link dependency, and how often
> users add includes to existing c++ files.
>

Right - my mental model was more that modules would be more one-to-one with
libraries, rather than the way headers are these days (with many headers
for one library). That seems to be incorrect & it's likely there may be
multiple modules (perhaps somewhat fewer than headers) for a single library.


> I expect you'll find the latter to be a far bigger number.
>
> I also expect that expecting users to edit their buildsystem, or allow it
> to be regenerated every time they add/remove includes would lead to less
> adoption of modules. I can see people trying them and then giving up in
> frustration.
>
> I think I read somewhere that the buildsystem in google already requires
> included '.h' files to be listed explicitly in the buildsystem, so it's no
> change in workflow there.
>

Not for the users of the library - but yes, a library needs to declare
which headers it contains/provides. (I'm not sure it's at the point where
that's required across the codebase - but it's a direction things are
moving in).


> For other teams, that would be a change which could be a change in
> workflow and something rebelled against.
>
> By the way, do you have any idea how much modules adoption would be needed
> to constitute "success"? Is there a goal there?
>

I don't know of any assessment like that.

> Nope, scratch that ^ I had thought that was the case, but talking more
> with Richard Smith it seems there's an expectation that modules will be
> somewhere between header and library granularity (obviously some small
> libraries today have one or only a few headers, some (like Qt) have many -
> maybe those on the Qt end might have slightly fewer modules than the have
> headers - but still several modules to one library most likely, by the
> sounds of it)
>
>
> Why? Richard maybe you can answer that? These are the kinds of things I
> was trying to get answers to in the previous post to iso sg2 in the google
> group. I didn't get an answer as definitive as this, so maybe you can share
> the reason behind such a definitive answer?
>

It's more that the functionality will allow this & just judging by how
people do things today (existing header granularity partly motivated by the
cost of headers that doesn't apply to modules), how they're likely to do
things in the future (I personally would guess people will probably try to
just port their headers to modules - and a few places where there are
circular dependencies in headers or the like they might glob them up into
one module).

> Now, admittedly, external dependencies are a little more complicated than
> internal (within a single project consisting of multiple libraries) - which
> is why I'd like to focus a bit on the simpler internal case first.
>
>
> Fair enough.
>
>
>
>
>> Today, a beginner can find a random C++ book, type in a code

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-05-07 Thread Brad King 
On 05/07/2018 12:01 PM, Stephen Kelly wrote:
> Hopefully Brad or someone else can provide other input from research already 
> done.

I'm not particularly familiar with what compiler writers or the modules
standard specification expects build systems to do w.r.t modules.
However, IIUC at least at one time the expectation was that the module
files would not be installed like headers and are used only within a
local build tree.  Are modules even supposed to be first-class entities
in the build system specification that users write?

In the Fortran world users just list all the sources and build systems are
expected to figure it out.  CMake has very good support for Fortran modules.
Our Ninja generator has rules to preprocess the translation units first,
then parse the preprocessed output to extract module definitions and usages,
then inject the added dependencies into the build graph, and then begin
compilation of sources ordered by those dependencies (this requires a
custom fork of Ninja pending upstream acceptance).

Is that what is expected from C++ buildsystems for modules too?

-Brad
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
https://cmake.org/mailman/listinfo/cmake-developers

Re: [cmake-developers] Experiments in CMake support for Clang (header & standard) modules

2018-05-07 Thread Stephen Kelly 

I think this discussion is more suited to the cmake-developers mailing
list. Moving there. Hopefully Brad or someone else can provide other
input from research already done.

On 05/07/2018 12:49 AM, David Blaikie wrote:
>
>> The basic commands required are:
>>
>>   clang++ -fmodules -xc++ -Xclang -emit-module -Xclang
>> -fmodules-codegen -fmodule-name=foo foo.modulemap -o foo.pcm
>>   clang++ -fmodules -c -fmodule-file=foo.pcm use.cpp
>>   clang++ -c foo.pcm
>>   clang++ foo.o use.o -o a.out
>
> Ok. Fundamentally, I am suspicious of having to have a
> -fmodule-file=foo.pcm for every 'import foo' in each cpp file. I
> shouldn't have to manually add that each time I add a new import
> to my cpp file. Even if it can be automated (eg by CMake), I
> shouldn't have to have my buildsystem be regenerated each time I
> add an import to my cpp file either.
>
> That's something I mentioned in the google groups post I made
> which you linked to. How will that work when using Qt or any other
> library?
>
>
> - My understanding/feeling is that this would be similar to how a user
> has to change their link command when they pick up a new dependency.

Perhaps it would be interesting to get an idea of how often users need
to change their buildsystems because of a new link dependency, and how
often users add includes to existing c++ files.

I expect you'll find the latter to be a far bigger number.

I also expect that expecting users to edit their buildsystem, or allow
it to be regenerated every time they add/remove includes would lead to
less adoption of modules. I can see people trying them and then giving
up in frustration.

I think I read somewhere that the buildsystem in google already requires
included '.h' files to be listed explicitly in the buildsystem, so it's
no change in workflow there. For other teams, that would be a change
which could be a change in workflow and something rebelled against.

By the way, do you have any idea how much modules adoption would be
needed to constitute "success"? Is there a goal there?

> Nope, scratch that ^ I had thought that was the case, but talking more
> with Richard Smith it seems there's an expectation that modules will
> be somewhere between header and library granularity (obviously some
> small libraries today have one or only a few headers, some (like Qt)
> have many - maybe those on the Qt end might have slightly fewer
> modules than the have headers - but still several modules to one
> library most likely, by the sounds of it)

Why? Richard maybe you can answer that? These are the kinds of things I
was trying to get answers to in the previous post to iso sg2 in the
google group. I didn't get an answer as definitive as this, so maybe you
can share the reason behind such a definitive answer?

> Now, admittedly, external dependencies are a little more complicated
> than internal (within a single project consisting of multiple
> libraries) - which is why I'd like to focus a bit on the simpler
> internal case first.

Fair enough.

>  
>
> Today, a beginner can find a random C++ book, type in a code
> example from chapter one and put `g++ -I/opt/book_examples
> prog1.cpp` into a terminal and get something compiling and
> running. With modules, they'll potentially have to pass a whole
> list of module files too.
>
>
> Yeah, there's some talk of supporting a mode that doesn't explicitly
> build/use modules in the filesystem, but only in memory for the
> purpose of preserving the isolation semantics of modules. This would
> be used in simple direct-compilation cases like this. Such a library
> might need a configuration file or similar the compiler can parse to
> discover the parameters (warning flags, define flags, whatever else)
> needed to build the BMI.

Perhaps. I'd be interested in how far into the book such a system would
take a beginner. Maybe that's fine, I don't know. Such a system might
not help with code in stack overflow questions/answers though, which
would probably be simpler sticking with includes (eg for Qt/boost).

Library authors will presumably have some say, or try to introduce some
'best practice' for users to follow. And such best practice will be
different for each library.
 
>  
>
> I raised some of these issues a few years ago regarding the clang
> implementation with files named exactly module.modulemap:
>
> 
> http://clang-developers.42468.n3.nabble.com/How-do-I-try-out-C-modules-with-clang-td4041946.html
>
> 
> http://clang-developers.42468.n3.nabble.com/How-do-I-try-out-C-modules-with-clang-td4041946i20.html
>
> Interestingly, GCC is taking a directory-centric approach in the
> driver (-fmodule-path=) as opposed to the 'add a file to your
> compile line for each import' that Clang and MSVC are taking:
>
>  http://gcc.gnu.org/wiki/cxx-modules
>
> Why is Clang not doing a directory-centric driver-interface? It
> seems to obviously