Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus,


2017-11-20 3:02 GMT+09:00 Linus Torvalds :
> On Sun, Nov 19, 2017 at 2:40 AM, Masahiro Yamada
>  wrote:
>>>
>>> FWIW, I still think we should probably make the compiler versions etc
>>> available to the configuration management rather than necessarily
>>> cache them.
>>
>> Do you mean something like this?
>>
>> https://lkml.org/lkml/2016/12/9/577
>
> Yes. With the important part not being so much that particular syntax,
> but the concept of "get some config options from automation".
>
> And the most obvious thing to do would be to just initialize a "bool"
> variable by running a script where the end result is either success or
> failure.
>
> Now, the actual example:
>
>config COMPILER_SUPPORTS_XYZ
>bool
>option shell="gcc -XYZ"
>
> is obviously too simplified to be realistic, because it would have to
> account for the actual compiler setup, so what you really need is not
> "just execute a shell command" but the equivalent of the Makefile
> "cc-option" function. Aes, for "known buggy versions" you might want
> to also get the actual compiler version into a config option.
>
>
>
> Even if "cc-option" is the _only_ thing you can do (and not some kind
> of "generic shell escape"), I think that would be very useful.
> Wouldn't it be nice to be able to have all those Makefile things as
> Kcconfig scripts - and be able to very naturally take them into
> account when offering people some Kconfig options?
>
> So you could do all the logic of not only testing what flags the
> compiler supports, but then use the Kconfig language to _combine_ that
> knowledge with the build options. Both in the sense of "this kernel
> config option depends on the compiler supporting flag XYZ" but also in
> the sense of "the use flag ABC depends on not only compiler support,
> but also on whether the kernel was configured for profiling" or
> whatever.
>
> Then the actual Makefile parts would be things like
>
> CFLAGS-$(CONFIG_CC_SPLIT_DWARF) += -gsplit_dwarf
>
> because the Kconfig phase would already have all the logic for whether
> (a) gcc actually supports -gsplit-dwarf at all and (b) whether the
> user actually asked for split debug info.
>
> And we'd not have the performance issues that made that whole caching
> thing be an issue, because the actual gcc support testing would only
> happen at Kconfig time.
>
> .. it would also make the caching rules be obvious. It's just a "if
> you upgraded or changed compilers enough to be noticeable, re-do your
> config ('make oldconfig')".
>
>Linus

OK, probably this is the right direction.

Embedded folks need to agree to give CROSS_COMPILE to the Kconfig phase.





-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus,


2017-11-20 3:02 GMT+09:00 Linus Torvalds :
> On Sun, Nov 19, 2017 at 2:40 AM, Masahiro Yamada
>  wrote:
>>>
>>> FWIW, I still think we should probably make the compiler versions etc
>>> available to the configuration management rather than necessarily
>>> cache them.
>>
>> Do you mean something like this?
>>
>> https://lkml.org/lkml/2016/12/9/577
>
> Yes. With the important part not being so much that particular syntax,
> but the concept of "get some config options from automation".
>
> And the most obvious thing to do would be to just initialize a "bool"
> variable by running a script where the end result is either success or
> failure.
>
> Now, the actual example:
>
>config COMPILER_SUPPORTS_XYZ
>bool
>option shell="gcc -XYZ"
>
> is obviously too simplified to be realistic, because it would have to
> account for the actual compiler setup, so what you really need is not
> "just execute a shell command" but the equivalent of the Makefile
> "cc-option" function. Aes, for "known buggy versions" you might want
> to also get the actual compiler version into a config option.
>
>
>
> Even if "cc-option" is the _only_ thing you can do (and not some kind
> of "generic shell escape"), I think that would be very useful.
> Wouldn't it be nice to be able to have all those Makefile things as
> Kcconfig scripts - and be able to very naturally take them into
> account when offering people some Kconfig options?
>
> So you could do all the logic of not only testing what flags the
> compiler supports, but then use the Kconfig language to _combine_ that
> knowledge with the build options. Both in the sense of "this kernel
> config option depends on the compiler supporting flag XYZ" but also in
> the sense of "the use flag ABC depends on not only compiler support,
> but also on whether the kernel was configured for profiling" or
> whatever.
>
> Then the actual Makefile parts would be things like
>
> CFLAGS-$(CONFIG_CC_SPLIT_DWARF) += -gsplit_dwarf
>
> because the Kconfig phase would already have all the logic for whether
> (a) gcc actually supports -gsplit-dwarf at all and (b) whether the
> user actually asked for split debug info.
>
> And we'd not have the performance issues that made that whole caching
> thing be an issue, because the actual gcc support testing would only
> happen at Kconfig time.
>
> .. it would also make the caching rules be obvious. It's just a "if
> you upgraded or changed compilers enough to be noticeable, re-do your
> config ('make oldconfig')".
>
>Linus

OK, probably this is the right direction.

Embedded folks need to agree to give CROSS_COMPILE to the Kconfig phase.





-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

On Sun, Nov 19, 2017 at 2:40 AM, Masahiro Yamada
 wrote:
>>
>> FWIW, I still think we should probably make the compiler versions etc
>> available to the configuration management rather than necessarily
>> cache them.
>
> Do you mean something like this?
>
> https://lkml.org/lkml/2016/12/9/577

Yes. With the important part not being so much that particular syntax,
but the concept of "get some config options from automation".

And the most obvious thing to do would be to just initialize a "bool"
variable by running a script where the end result is either success or
failure.

Now, the actual example:

   config COMPILER_SUPPORTS_XYZ
   bool
   option shell="gcc -XYZ"

is obviously too simplified to be realistic, because it would have to
account for the actual compiler setup, so what you really need is not
"just execute a shell command" but the equivalent of the Makefile
"cc-option" function. Aes, for "known buggy versions" you might want
to also get the actual compiler version into a config option.



Even if "cc-option" is the _only_ thing you can do (and not some kind
of "generic shell escape"), I think that would be very useful.
Wouldn't it be nice to be able to have all those Makefile things as
Kcconfig scripts - and be able to very naturally take them into
account when offering people some Kconfig options?

So you could do all the logic of not only testing what flags the
compiler supports, but then use the Kconfig language to _combine_ that
knowledge with the build options. Both in the sense of "this kernel
config option depends on the compiler supporting flag XYZ" but also in
the sense of "the use flag ABC depends on not only compiler support,
but also on whether the kernel was configured for profiling" or
whatever.

Then the actual Makefile parts would be things like

CFLAGS-$(CONFIG_CC_SPLIT_DWARF) += -gsplit_dwarf

because the Kconfig phase would already have all the logic for whether
(a) gcc actually supports -gsplit-dwarf at all and (b) whether the
user actually asked for split debug info.

And we'd not have the performance issues that made that whole caching
thing be an issue, because the actual gcc support testing would only
happen at Kconfig time.

.. it would also make the caching rules be obvious. It's just a "if
you upgraded or changed compilers enough to be noticeable, re-do your
config ('make oldconfig')".

   Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

On Sun, Nov 19, 2017 at 2:40 AM, Masahiro Yamada
 wrote:
>>
>> FWIW, I still think we should probably make the compiler versions etc
>> available to the configuration management rather than necessarily
>> cache them.
>
> Do you mean something like this?
>
> https://lkml.org/lkml/2016/12/9/577

Yes. With the important part not being so much that particular syntax,
but the concept of "get some config options from automation".

And the most obvious thing to do would be to just initialize a "bool"
variable by running a script where the end result is either success or
failure.

Now, the actual example:

   config COMPILER_SUPPORTS_XYZ
   bool
   option shell="gcc -XYZ"

is obviously too simplified to be realistic, because it would have to
account for the actual compiler setup, so what you really need is not
"just execute a shell command" but the equivalent of the Makefile
"cc-option" function. Aes, for "known buggy versions" you might want
to also get the actual compiler version into a config option.



Even if "cc-option" is the _only_ thing you can do (and not some kind
of "generic shell escape"), I think that would be very useful.
Wouldn't it be nice to be able to have all those Makefile things as
Kcconfig scripts - and be able to very naturally take them into
account when offering people some Kconfig options?

So you could do all the logic of not only testing what flags the
compiler supports, but then use the Kconfig language to _combine_ that
knowledge with the build options. Both in the sense of "this kernel
config option depends on the compiler supporting flag XYZ" but also in
the sense of "the use flag ABC depends on not only compiler support,
but also on whether the kernel was configured for profiling" or
whatever.

Then the actual Makefile parts would be things like

CFLAGS-$(CONFIG_CC_SPLIT_DWARF) += -gsplit_dwarf

because the Kconfig phase would already have all the logic for whether
(a) gcc actually supports -gsplit-dwarf at all and (b) whether the
user actually asked for split debug info.

And we'd not have the performance issues that made that whole caching
thing be an issue, because the actual gcc support testing would only
happen at Kconfig time.

.. it would also make the caching rules be obvious. It's just a "if
you upgraded or changed compilers enough to be noticeable, re-do your
config ('make oldconfig')".

   Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus


2017-11-18 11:01 GMT+09:00 Linus Torvalds :
> Oh, and I forgot to ask..
>
> On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
>  wrote:
>>
>> One of the most remarkable improvements in this cycle is, Kbuild is
>> now able to cache the result of shell commands.
>
> I see the "limit it to 500 lines", but I don't see any real coherency.

The limit is 1000 lines.
If your cache file exceeds 1000 lines,
it will be cut down to 500 lines at the next invocation of build.

I used two values 1000 and 500
so that the cache shrink operation is not triggered every time.


> So I take it that if you upgrade your gcc version, you may need to
> blow this cache away manually?

Right.  This is a limitation of this feature.
But, this limitation has existed since before.

When you upgrade your gcc,
you need to do "make clean" anyway to blow all *.o files
so that all objects are re-compiled by the new gcc.

Kbuild stores build commands in .*.cmd files,
but it cannot notice the compiler upgrade.



> Or is there something subtle going on that I've missed?
>
> FWIW, I still think we should probably make the compiler versions etc
> available to the configuration management rather than necessarily
> cache them.


Do you mean something like this?

https://lkml.org/lkml/2016/12/9/577



At first, I thought it was allowed to use a different compiler
for external modules than the one compiled the kernel.
But, Greg said we do not support that case.

Then, the runtime test of compiler capabilities is pointless,
so I think it is a possible solution.
CONFIG_CC_STACKPROTECTOR_AUTO will even more mess up the top Makefile.
https://patchwork.kernel.org/patch/9981173/


-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus


2017-11-18 11:01 GMT+09:00 Linus Torvalds :
> Oh, and I forgot to ask..
>
> On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
>  wrote:
>>
>> One of the most remarkable improvements in this cycle is, Kbuild is
>> now able to cache the result of shell commands.
>
> I see the "limit it to 500 lines", but I don't see any real coherency.

The limit is 1000 lines.
If your cache file exceeds 1000 lines,
it will be cut down to 500 lines at the next invocation of build.

I used two values 1000 and 500
so that the cache shrink operation is not triggered every time.


> So I take it that if you upgrade your gcc version, you may need to
> blow this cache away manually?

Right.  This is a limitation of this feature.
But, this limitation has existed since before.

When you upgrade your gcc,
you need to do "make clean" anyway to blow all *.o files
so that all objects are re-compiled by the new gcc.

Kbuild stores build commands in .*.cmd files,
but it cannot notice the compiler upgrade.



> Or is there something subtle going on that I've missed?
>
> FWIW, I still think we should probably make the compiler versions etc
> available to the configuration management rather than necessarily
> cache them.


Do you mean something like this?

https://lkml.org/lkml/2016/12/9/577



At first, I thought it was allowed to use a different compiler
for external modules than the one compiled the kernel.
But, Greg said we do not support that case.

Then, the runtime test of compiler capabilities is pointless,
so I think it is a possible solution.
CONFIG_CC_STACKPROTECTOR_AUTO will even more mess up the top Makefile.
https://patchwork.kernel.org/patch/9981173/


-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus,


2017-11-18 10:51 GMT+09:00 Linus Torvalds :
> On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
>  wrote:
>>
>> Here are Kbuild updates for v4.15.  Please pull!
>
> Mind checking that I got the CFLAGS_KCOV behavior right. The conflict
> looked pretty simple, but somebody should check the ordering of the
>
>export CFLAGS_KCOV
>
> vs the actual setting (that is now in scripts/Makefile.kcov.
>
> It probably doesn't matter, but our makefiles are black magic.
>
>   Linus


Strictly speaking, there is a case where the placement of "export"
make a difference.


-(example1)-
BAR := bar

FOO += $(BAR)
export FOO

BAR := bar2
-


-(example2)-
BAR := bar

export FOO
FOO += $(BAR)

BAR := bar2
-


In example1, the exported value for FOO is "bar2"
whereas, in example2, "bar".

The difference probably comes "export" implies
":=" flavor initialization if not explicitly initialized yet.
To avoid confusion, we should explicitly use either ":=" or "=", though.


Talking about the kcov case, CFLAGS_KCOV is initialized by ":="
in scripts/Malefile.kcov, so it should work as we expect.

(In my personal preference, I'd like collect
the kcov stuff into scripts/Makefile.kcof, but
it is just a matter of slight taste.)



-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Masahiro Yamada]

Hi Linus,


2017-11-18 10:51 GMT+09:00 Linus Torvalds :
> On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
>  wrote:
>>
>> Here are Kbuild updates for v4.15.  Please pull!
>
> Mind checking that I got the CFLAGS_KCOV behavior right. The conflict
> looked pretty simple, but somebody should check the ordering of the
>
>export CFLAGS_KCOV
>
> vs the actual setting (that is now in scripts/Makefile.kcov.
>
> It probably doesn't matter, but our makefiles are black magic.
>
>   Linus


Strictly speaking, there is a case where the placement of "export"
make a difference.


-(example1)-
BAR := bar

FOO += $(BAR)
export FOO

BAR := bar2
-


-(example2)-
BAR := bar

export FOO
FOO += $(BAR)

BAR := bar2
-


In example1, the exported value for FOO is "bar2"
whereas, in example2, "bar".

The difference probably comes "export" implies
":=" flavor initialization if not explicitly initialized yet.
To avoid confusion, we should explicitly use either ":=" or "=", though.


Talking about the kcov case, CFLAGS_KCOV is initialized by ":="
in scripts/Malefile.kcov, so it should work as we expect.

(In my personal preference, I'd like collect
the kcov stuff into scripts/Makefile.kcof, but
it is just a matter of slight taste.)



-- 
Best Regards
Masahiro Yamada

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

Oh, and I forgot to ask..

On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
 wrote:
>
> One of the most remarkable improvements in this cycle is, Kbuild is
> now able to cache the result of shell commands.

I see the "limit it to 500 lines", but I don't see any real coherency.

So I take it that if you upgrade your gcc version, you may need to
blow this cache away manually?

Or is there something subtle going on that I've missed?

FWIW, I still think we should probably make the compiler versions etc
available to the configuration management rather than necessarily
cache them.

Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

Oh, and I forgot to ask..

On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
 wrote:
>
> One of the most remarkable improvements in this cycle is, Kbuild is
> now able to cache the result of shell commands.

I see the "limit it to 500 lines", but I don't see any real coherency.

So I take it that if you upgrade your gcc version, you may need to
blow this cache away manually?

Or is there something subtle going on that I've missed?

FWIW, I still think we should probably make the compiler versions etc
available to the configuration management rather than necessarily
cache them.

Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
 wrote:
>
> Here are Kbuild updates for v4.15.  Please pull!

Mind checking that I got the CFLAGS_KCOV behavior right. The conflict
looked pretty simple, but somebody should check the ordering of the

   export CFLAGS_KCOV

vs the actual setting (that is now in scripts/Makefile.kcov.

It probably doesn't matter, but our makefiles are black magic.

  Linus

Re: [GIT PULL 1/2] Kbuild updates for v4.15

[Linus Torvalds]

On Fri, Nov 17, 2017 at 9:22 AM, Masahiro Yamada
 wrote:
>
> Here are Kbuild updates for v4.15.  Please pull!

Mind checking that I got the CFLAGS_KCOV behavior right. The conflict
looked pretty simple, but somebody should check the ordering of the

   export CFLAGS_KCOV

vs the actual setting (that is now in scripts/Makefile.kcov.

It probably doesn't matter, but our makefiles are black magic.

  Linus