Re: [Xen-devel] [PATCH v4 08/16] kbuild: enable option to force compile force-obj-y and force-lib-y

[Luis R. Rodriguez]

On Tue, Aug 23, 2016 at 01:59:10AM +0200, Luis R. Rodriguez wrote:
> On Fri, Aug 19, 2016 at 03:10:33PM -0700, Kees Cook wrote:
> > On Fri, Aug 19, 2016 at 2:32 PM,   wrote:
> > > diff --git a/init/Kconfig b/init/Kconfig
> > > index cac3f096050d..ef09e83b9196 100644
> > > --- a/init/Kconfig
> > > +++ b/init/Kconfig
> > > @@ -53,6 +53,28 @@ config CROSS_COMPILE
> > >   need to set this unless you want the configured kernel build
> > >   directory to select the cross-compiler automatically.
> > >
> > > +config BUILD_AVOID_BITROT
> > > +   bool "Enable force building of force-obj-y and force-lib-y"
> > 
> > Sorry to continue the bikeshedding on this, but if I encounter
> > something in a Makefile named "force-obj-y" I would expect it to
> > always be built, no matter what. But this is not the case: the
> > "force-" prefix is tied to this CONFIG_BUILD_AVOID_BITROT. This verb
> > usage is weird, as I'd expect an adjective, like "forceable-obj-y" or
> > something that describes that it CAN be built even with the CONFIG for
> > it is missing, etc.
> > 
> > Regardless, I defer to Michal on this, but I'm not a fan of "force"
> > being used when it's an optional action. :)
> 
> Sure, Michal let me know if forceable-obj-y and forceable-lib-y is the
> way to go.

Michal? Any preference ? Or how about opt-force-obj ?

  Luis

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 08/16] kbuild: enable option to force compile force-obj-y and force-lib-y

[Luis R. Rodriguez]

On Fri, Aug 19, 2016 at 03:10:33PM -0700, Kees Cook wrote:
> On Fri, Aug 19, 2016 at 2:32 PM,   wrote:
> > diff --git a/init/Kconfig b/init/Kconfig
> > index cac3f096050d..ef09e83b9196 100644
> > --- a/init/Kconfig
> > +++ b/init/Kconfig
> > @@ -53,6 +53,28 @@ config CROSS_COMPILE
> >   need to set this unless you want the configured kernel build
> >   directory to select the cross-compiler automatically.
> >
> > +config BUILD_AVOID_BITROT
> > +   bool "Enable force building of force-obj-y and force-lib-y"
> 
> Sorry to continue the bikeshedding on this, but if I encounter
> something in a Makefile named "force-obj-y" I would expect it to
> always be built, no matter what. But this is not the case: the
> "force-" prefix is tied to this CONFIG_BUILD_AVOID_BITROT. This verb
> usage is weird, as I'd expect an adjective, like "forceable-obj-y" or
> something that describes that it CAN be built even with the CONFIG for
> it is missing, etc.
> 
> Regardless, I defer to Michal on this, but I'm not a fan of "force"
> being used when it's an optional action. :)

Sure, Michal let me know if forceable-obj-y and forceable-lib-y is the
way to go.

  Luis

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v4 08/16] kbuild: enable option to force compile force-obj-y and force-lib-y

[Kees Cook]

On Fri, Aug 19, 2016 at 2:32 PM,   wrote:
> From: "Luis R. Rodriguez" 
>
> Linux provides a rich array of features, enabling each feature
> however increases the size of the kernel and there are many
> features which users often want disabled. The traditional
> solution to this problem is for each feature to have its own
> Kconfig symbol, followed by a series of #ifdef statements
> in C code and header files, allowing the feature to be compiled
> only when desirable. As the variability of Linux increases build
> tests can and are often done with random kernel configurations,
> allyesconfig, and allmodconfig to help find code issues. This
> however doesn't catch all errors and as a consequence code that
> is typically not enabled often can suffer from bit-rot over time.
>
> An alternative approach for subsystems, which refer to as the 'build-all
> link-selectively philosophy' is to keep the Kconfig symbols, replace
> the #ifdef approach by having each feature implemented it its own C file,
> and force compilation for all features to avoid the code bit-rot problem.
> With this strategy only features that are enabled via Kconfig get
> linked into the kernel, so the forced compilation has no size impact
> on the kernel. The practice of having each feature implemented in its own
> C file is already prevalent in many subsystems, however #ifdefs are still
> typically required during feature initialization. For instance in:
>
>   #ifdef CONFIG_FOO
>   foo_init();
>   #endif
>
> We cannot remove the #ifdef and leave foo_init() as we'd either
> need to always enable the feature or add a respective #ifdef in a
> foo.h which makes foo_init() do nothing when CONFIG_FOO is disabled.
>
> Linker tables enable lifting the requirement to use of #ifdefs during
> initialization. With linker tables initialization sequences can instead
> be aggregated into a custom ELF section at link time, during run time
> the table can be iterated over and each init sequence enabled can be called.
> A feature's init routine is only added to a table when its respective
> Kconfig symbols has been enabled and therefore linked in. Linker tables
> enable subsystems to completely do away with #ifdefs if one is comfortable
> in accepting all subsystem's feature's structural size implications.
>
> Subsystems which want to follow the 'build-all link-selectively
> philosophy' still need a way to easily express and annotate that they
> wish for all code to always be compiled to help avoid code bit rot,
> as such two new targets force-obj-y and force-lib-y are provided to
> help with this. Its not fair to require everyone to force compilation
> of all features of a subsystem though, so as a compromise, the new
> targets only force compilation when CONFIG_BUILD_AVOID_BITROT is
> enabled.
>
> Only built-in features are supported at the moment. Module support
> is expected to be added after a generic solution to add linker
> tables to modules more easily is developed.
>
> v4: this patch was added to this series, it was split off from the
> linker tables addition due to the confusion over the code bit
> rot alternatives that are possible with linker tables.
>
> Signed-off-by: Luis R. Rodriguez 
> ---
>  Documentation/kbuild/makefiles.txt   | 36 
>  Documentation/sections/linker-tables.rst | 15 +++
>  include/linux/tables.h   | 71 
> 
>  init/Kconfig | 22 ++
>  scripts/Makefile.build   |  7 ++--
>  scripts/Makefile.lib | 11 +
>  6 files changed, 159 insertions(+), 3 deletions(-)
>
> diff --git a/Documentation/kbuild/makefiles.txt 
> b/Documentation/kbuild/makefiles.txt
> index 385a5ef41c17..01c260913f5c 100644
> --- a/Documentation/kbuild/makefiles.txt
> +++ b/Documentation/kbuild/makefiles.txt
> @@ -1089,6 +1089,42 @@ When kbuild executes, the following steps are followed 
> (roughly):
> In this example, extra-y is used to list object files that
> shall be built, but shall not be linked as part of built-in.o.
>
> +force-obj-y force-lib-y
> +
> +   When CONFIG_BUILD_AVOID_BITROT is enabled using these targets for your
> +   kconfig symbols forces compilation of the associated objects if the
> +   kconfig's symbol's dependencies are met, the objects however are only
> +   linked into to the kernel if and only if the kconfig symbol was
> +   enabled. If CONFIG_BUILD_AVOID_BITROT is disabled the force-obj-y and
> +   force-lib-y targets are functionally equilvalent to obj-y and lib-y
> +   respectively.
> +
> +   Using force-obj-y and force-lib-y are part of a code architecture and
> +   build philosophy further enabled by linker tables, for more details
> +   refer to the documention in include/linux/tables.h, refer to the
> +   sections:
> +
> +   o The code bit-rot problem

[Xen-devel] [PATCH v4 08/16] kbuild: enable option to force compile force-obj-y and force-lib-y

[mcgrof]

From: "Luis R. Rodriguez" 

Linux provides a rich array of features, enabling each feature
however increases the size of the kernel and there are many
features which users often want disabled. The traditional
solution to this problem is for each feature to have its own
Kconfig symbol, followed by a series of #ifdef statements
in C code and header files, allowing the feature to be compiled
only when desirable. As the variability of Linux increases build
tests can and are often done with random kernel configurations,
allyesconfig, and allmodconfig to help find code issues. This
however doesn't catch all errors and as a consequence code that
is typically not enabled often can suffer from bit-rot over time.

An alternative approach for subsystems, which refer to as the 'build-all
link-selectively philosophy' is to keep the Kconfig symbols, replace
the #ifdef approach by having each feature implemented it its own C file,
and force compilation for all features to avoid the code bit-rot problem.
With this strategy only features that are enabled via Kconfig get
linked into the kernel, so the forced compilation has no size impact
on the kernel. The practice of having each feature implemented in its own
C file is already prevalent in many subsystems, however #ifdefs are still
typically required during feature initialization. For instance in:

  #ifdef CONFIG_FOO
  foo_init();
  #endif

We cannot remove the #ifdef and leave foo_init() as we'd either
need to always enable the feature or add a respective #ifdef in a
foo.h which makes foo_init() do nothing when CONFIG_FOO is disabled.

Linker tables enable lifting the requirement to use of #ifdefs during
initialization. With linker tables initialization sequences can instead
be aggregated into a custom ELF section at link time, during run time
the table can be iterated over and each init sequence enabled can be called.
A feature's init routine is only added to a table when its respective
Kconfig symbols has been enabled and therefore linked in. Linker tables
enable subsystems to completely do away with #ifdefs if one is comfortable
in accepting all subsystem's feature's structural size implications.

Subsystems which want to follow the 'build-all link-selectively
philosophy' still need a way to easily express and annotate that they
wish for all code to always be compiled to help avoid code bit rot,
as such two new targets force-obj-y and force-lib-y are provided to
help with this. Its not fair to require everyone to force compilation
of all features of a subsystem though, so as a compromise, the new
targets only force compilation when CONFIG_BUILD_AVOID_BITROT is
enabled.

Only built-in features are supported at the moment. Module support
is expected to be added after a generic solution to add linker
tables to modules more easily is developed.

v4: this patch was added to this series, it was split off from the
linker tables addition due to the confusion over the code bit
rot alternatives that are possible with linker tables.

Signed-off-by: Luis R. Rodriguez 
---
 Documentation/kbuild/makefiles.txt   | 36 
 Documentation/sections/linker-tables.rst | 15 +++
 include/linux/tables.h   | 71 
 init/Kconfig | 22 ++
 scripts/Makefile.build   |  7 ++--
 scripts/Makefile.lib | 11 +
 6 files changed, 159 insertions(+), 3 deletions(-)

diff --git a/Documentation/kbuild/makefiles.txt 
b/Documentation/kbuild/makefiles.txt
index 385a5ef41c17..01c260913f5c 100644
--- a/Documentation/kbuild/makefiles.txt
+++ b/Documentation/kbuild/makefiles.txt
@@ -1089,6 +1089,42 @@ When kbuild executes, the following steps are followed 
(roughly):
In this example, extra-y is used to list object files that
shall be built, but shall not be linked as part of built-in.o.
 
+force-obj-y force-lib-y
+
+   When CONFIG_BUILD_AVOID_BITROT is enabled using these targets for your
+   kconfig symbols forces compilation of the associated objects if the
+   kconfig's symbol's dependencies are met, the objects however are only
+   linked into to the kernel if and only if the kconfig symbol was
+   enabled. If CONFIG_BUILD_AVOID_BITROT is disabled the force-obj-y and
+   force-lib-y targets are functionally equilvalent to obj-y and lib-y
+   respectively.
+
+   Using force-obj-y and force-lib-y are part of a code architecture and
+   build philosophy further enabled by linker tables, for more details
+   refer to the documention in include/linux/tables.h, refer to the
+   sections:
+
+   o The code bit-rot problem
+   o The build-all selective-link philosophy
+   o Avoiding the code bit-rot problem with linker tables
+   o Linker table module support
+
+   Modules support is expected to be enhanced in the 

[Xen-devel] [PATCH v4 08/16] kbuild: enable option to force compile force-obj-y and force-lib-y

[mcgrof]

From: "Luis R. Rodriguez" 

Linux provides a rich array of features, enabling each feature
however increases the size of the kernel and there are many
features which users often want disabled. The traditional
solution to this problem is for each feature to have its own
Kconfig symbol, followed by a series of #ifdef statements
in C code and header files, allowing the feature to be compiled
only when desirable. As the variability of Linux increases build
tests can and are often done with random kernel configurations,
allyesconfig, and allmodconfig to help find code issues. This
however doesn't catch all errors and as a consequence code that
is typically not enabled often can suffer from bit-rot over time.

An alternative approach for subsystems, which refer to as the 'build-all
link-selectively philosophy' is to keep the Kconfig symbols, replace
the #ifdef approach by having each feature implemented it its own C file,
and force compilation for all features to avoid the code bit-rot problem.
With this strategy only features that are enabled via Kconfig get
linked into the kernel, so the forced compilation has no size impact
on the kernel. The practice of having each feature implemented in its own
C file is already prevalent in many subsystems, however #ifdefs are still
typically required during feature initialization. For instance in:

  #ifdef CONFIG_FOO
  foo_init();
  #endif

We cannot remove the #ifdef and leave foo_init() as we'd either
need to always enable the feature or add a respective #ifdef in a
foo.h which makes foo_init() do nothing when CONFIG_FOO is disabled.

Linker tables enable lifting the requirement to use of #ifdefs during
initialization. With linker tables initialization sequences can instead
be aggregated into a custom ELF section at link time, during run time
the table can be iterated over and each init sequence enabled can be called.
A feature's init routine is only added to a table when its respective
Kconfig symbols has been enabled and therefore linked in. Linker tables
enable subsystems to completely do away with #ifdefs if one is comfortable
in accepting all subsystem's feature's structural size implications.

Subsystems which want to follow the 'build-all link-selectively
philosophy' still need a way to easily express and annotate that they
wish for all code to always be compiled to help avoid code bit rot,
as such two new targets force-obj-y and force-lib-y are provided to
help with this. Its not fair to require everyone to force compilation
of all features of a subsystem though, so as a compromise, the new
targets only force compilation when CONFIG_BUILD_AVOID_BITROT is
enabled.

Only built-in features are supported at the moment. Module support
is expected to be added after a generic solution to add linker
tables to modules more easily is developed.

v4: this patch was added to this series, it was split off from the
linker tables addition due to the confusion over the code bit
rot alternatives that are possible with linker tables.

Signed-off-by: Luis R. Rodriguez 
---
 Documentation/kbuild/makefiles.txt   | 36 
 Documentation/sections/linker-tables.rst | 15 +++
 include/linux/tables.h   | 71 
 init/Kconfig | 22 ++
 scripts/Makefile.build   |  7 ++--
 scripts/Makefile.lib | 11 +
 6 files changed, 159 insertions(+), 3 deletions(-)

diff --git a/Documentation/kbuild/makefiles.txt 
b/Documentation/kbuild/makefiles.txt
index 385a5ef41c17..01c260913f5c 100644
--- a/Documentation/kbuild/makefiles.txt
+++ b/Documentation/kbuild/makefiles.txt
@@ -1089,6 +1089,42 @@ When kbuild executes, the following steps are followed 
(roughly):
In this example, extra-y is used to list object files that
shall be built, but shall not be linked as part of built-in.o.
 
+force-obj-y force-lib-y
+
+   When CONFIG_BUILD_AVOID_BITROT is enabled using these targets for your
+   kconfig symbols forces compilation of the associated objects if the
+   kconfig's symbol's dependencies are met, the objects however are only
+   linked into to the kernel if and only if the kconfig symbol was
+   enabled. If CONFIG_BUILD_AVOID_BITROT is disabled the force-obj-y and
+   force-lib-y targets are functionally equilvalent to obj-y and lib-y
+   respectively.
+
+   Using force-obj-y and force-lib-y are part of a code architecture and
+   build philosophy further enabled by linker tables, for more details
+   refer to the documention in include/linux/tables.h, refer to the
+   sections:
+
+   o The code bit-rot problem
+   o The build-all selective-link philosophy
+   o Avoiding the code bit-rot problem with linker tables
+   o Linker table module support
+
+   Modules support is expected to be enhanced in the