Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Bruce Ashfield]

On Wed, Apr 20, 2022 at 3:23 PM Jimmy Assarsson 
wrote:

> On 2022-03-13 04:37, Bruce Ashfield wrote:
> > On Sat, Mar 12, 2022 at 2:12 PM Jimmy Assarsson <
> jimmyassars...@gmail.com > wrote:
> >
> > On 2022-03-09 16:47, Bruce Ashfield wrote:
> >  > On Sat, Feb 26, 2022 at 2:17 AM Jimmy Assarsson
> >  > mailto:jimmyassars...@gmail.com>>
> wrote:
> >  >> On 2022-02-24 16:41, Bruce Ashfield wrote:
> >  >>> On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson
> >  >>> mailto:jimmyassars...@gmail.com>>
> wrote:
> >   On 2022-02-21 05:13, Bruce Ashfield wrote:
> >  > on 19/02/2022 Jimmy Assarsson wrote:
> >  >> On 2022-02-19 20:42, Bruce Ashfield wrote:
> >  >>> On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
> >  >>> mailto:jimmyassars...@gmail.com>>
> wrote:
> >  
> >   When patching a kernel git repository, in a recipe
> inheriting kernel-yocto.bbclass,
> >   the resulting commit hash will become different every time
> the source is unpacked and patched.
> >  
> >   This is a problem that causes non-reproducible builds,
> when the commit hash is built
> >   into the kernel (CONFIG_LOCALVERSION_AUTO=y).
> >  
> >  
> >   Currently it is not a problem in linux-yocto, since an
> empty .scmversion is
> >   created in kernel_do_configure [1]. This is preventing the
> kernel build from
> >   generating its own .scmversion.
> >  
> >   If removing this commit, setting
> CONFIG_LOCALVERSION_AUTO=y and applying any
> >   patch in the linux-yocto recipe, this will result in a
> non-reproducible build.
> >  >>
> >  >> ...
> >  >>
> >   diff --git a/tools/kgit-s2q b/tools/kgit-s2q
> >   index 706783e..b46a138 100755
> >   --- a/tools/kgit-s2q
> >   +++ b/tools/kgit-s2q
> >   @@ -558,7 +558,7 @@ do
> >    echo "($APPLIED/$COUNT) `basename $i`"
> >    fi
> >  
> >   -   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
> >   +   git am --keep-non-patch
> --committer-date-is-author-date $DIR/$i > /dev/null 2>&1
> >    if [ $? != 0 ];then
> >    git am --abort > /dev/null 2>&1
> >    echo "[INFO]: check of $DIR/$i with
> \"git am\" did not pass, trying reduced context."
> >  
> >  
> >  
> >   I'm not sure if this is a proper solution to fix the
> problem or what side effects it may introduce?
> >  >>>
> >  >>> There's nothing fundamentally wrong with that solution, but
> there are
> >  >>> modes for the kernel builds where we absolutely do not want
> the
> >  >>> reproducible build (and older dates) to be in effect (see
> my comments
> >  >>> on the OE core mailing list when the ability to disable
> reproducible
> >  >>> builds was almost removed).
> >  >>
> >  >> Thanks, now I've looked it up.
> >  >>
> >  >>> We could add an option to the patch queue management that
> was enabled
> >  >>> when reproducible builds are in play, so git am could
> operate like
> >  >>> that.
> >  >>>
> >  >>> Alternatively, you could bbappend the other kernel's
> do_configure and
> >  >>> do the same .scmversion trick to make sure that everything
> is
> >  >>> consistent.
> >  >>
> >  >> I don't follow, can you be more specific?
> >  >> We encountered this problem when building linux-imx (from
> meta-freescale).
> >  >> Preferably the .scmversion workaround in [1] should be
> dropped, so that
> >  >> there is no need to reinvent/mimic the output of
> >  >> linux/scripts/setlocalversion, in a bitbake recipe.
> >  >
> >  > I see no compelling reason to drop the .scmversion in the
> main kernel.bbclass
> >  > (at least not in the short term). It has been there for a
> significant amount
> >  > of time, and removing it now would very likely cause some
> ripple effects
> >  > througout the ecosystem (it is there for a few different
> issues).  While not
> >  > elegant, it is an available and workable solution to the
> problem it is
> >  > solving.
> >  
> >   I agree. And obviously this is not a big issue for others.
> >   There have been at least two failed attempts on removing this
> [4][5][6].
> >  
> >  > I was suggesting that anyone can bbappend any kernel recipe
> to do the similar
> >  > .scmversion set (assuming they aren't using the core kernel
> bbclass
> >  > configure()).
> >  

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Jimmy Assarsson]

On 2022-03-13 04:37, Bruce Ashfield wrote:

On Sat, Mar 12, 2022 at 2:12 PM Jimmy Assarsson mailto:jimmyassars...@gmail.com>> wrote:

On 2022-03-09 16:47, Bruce Ashfield wrote:
 > On Sat, Feb 26, 2022 at 2:17 AM Jimmy Assarsson
 > mailto:jimmyassars...@gmail.com>> wrote:
 >> On 2022-02-24 16:41, Bruce Ashfield wrote:
 >>> On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson
 >>> mailto:jimmyassars...@gmail.com>> wrote:
  On 2022-02-21 05:13, Bruce Ashfield wrote:
 > on 19/02/2022 Jimmy Assarsson wrote:
 >> On 2022-02-19 20:42, Bruce Ashfield wrote:
 >>> On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
 >>> mailto:jimmyassars...@gmail.com>> wrote:
 
  When patching a kernel git repository, in a recipe inheriting 
kernel-yocto.bbclass,
  the resulting commit hash will become different every time the 
source is unpacked and patched.
 
  This is a problem that causes non-reproducible builds, when the 
commit hash is built
  into the kernel (CONFIG_LOCALVERSION_AUTO=y).
 
 
  Currently it is not a problem in linux-yocto, since an empty 
.scmversion is
  created in kernel_do_configure [1]. This is preventing the kernel 
build from
  generating its own .scmversion.
 
  If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and 
applying any
  patch in the linux-yocto recipe, this will result in a 
non-reproducible build.
 >>
 >> ...
 >>
  diff --git a/tools/kgit-s2q b/tools/kgit-s2q
  index 706783e..b46a138 100755
  --- a/tools/kgit-s2q
  +++ b/tools/kgit-s2q
  @@ -558,7 +558,7 @@ do
                       echo "($APPLIED/$COUNT) `basename $i`"
               fi
 
  -       git am --keep-non-patch $DIR/$i > /dev/null 2>&1
  +       git am --keep-non-patch --committer-date-is-author-date $DIR/$i 
> /dev/null 2>&1
               if [ $? != 0 ];then
                       git am --abort > /dev/null 2>&1
                       echo "[INFO]: check of $DIR/$i with \"git am\" 
did not pass, trying reduced context."
 
 
 
  I'm not sure if this is a proper solution to fix the problem or 
what side effects it may introduce?
 >>>
 >>> There's nothing fundamentally wrong with that solution, but there 
are
 >>> modes for the kernel builds where we absolutely do not want the
 >>> reproducible build (and older dates) to be in effect (see my 
comments
 >>> on the OE core mailing list when the ability to disable 
reproducible
 >>> builds was almost removed).
 >>
 >> Thanks, now I've looked it up.
 >>
 >>> We could add an option to the patch queue management that was 
enabled
 >>> when reproducible builds are in play, so git am could operate like
 >>> that.
 >>>
 >>> Alternatively, you could bbappend the other kernel's do_configure 
and
 >>> do the same .scmversion trick to make sure that everything is
 >>> consistent.
 >>
 >> I don't follow, can you be more specific?
 >> We encountered this problem when building linux-imx (from 
meta-freescale).
 >> Preferably the .scmversion workaround in [1] should be dropped, so 
that
 >> there is no need to reinvent/mimic the output of
 >> linux/scripts/setlocalversion, in a bitbake recipe.
 >
 > I see no compelling reason to drop the .scmversion in the main 
kernel.bbclass
 > (at least not in the short term). It has been there for a 
significant amount
 > of time, and removing it now would very likely cause some ripple 
effects
 > througout the ecosystem (it is there for a few different issues).  
While not
 > elegant, it is an available and workable solution to the problem it 
is
 > solving.
 
  I agree. And obviously this is not a big issue for others.
  There have been at least two failed attempts on removing this 
[4][5][6].
 
 > I was suggesting that anyone can bbappend any kernel recipe to do 
the similar
 > .scmversion set (assuming they aren't using the core kernel bbclass
 > configure()).
 >
 > But from your next statement, that isn't going to work as the sole 
solution,
 > since yes, that will only restore where we keep the git rev out of 
the
 > localversion.
 >
 >>
 >> To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
 >> with a reproducible output.
 >
 > And that's actually part of the reason the .scmversion is set, the
 > setlocalversion 

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Bruce Ashfield]

On Sat, Mar 12, 2022 at 2:12 PM Jimmy Assarsson 
wrote:

> On 2022-03-09 16:47, Bruce Ashfield wrote:
> > On Sat, Feb 26, 2022 at 2:17 AM Jimmy Assarsson
> >  wrote:
> >> On 2022-02-24 16:41, Bruce Ashfield wrote:
> >>> On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson
> >>>  wrote:
>  On 2022-02-21 05:13, Bruce Ashfield wrote:
> > on 19/02/2022 Jimmy Assarsson wrote:
> >> On 2022-02-19 20:42, Bruce Ashfield wrote:
> >>> On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
> >>>  wrote:
> 
>  When patching a kernel git repository, in a recipe inheriting
> kernel-yocto.bbclass,
>  the resulting commit hash will become different every time the
> source is unpacked and patched.
> 
>  This is a problem that causes non-reproducible builds, when the
> commit hash is built
>  into the kernel (CONFIG_LOCALVERSION_AUTO=y).
> 
> 
>  Currently it is not a problem in linux-yocto, since an empty
> .scmversion is
>  created in kernel_do_configure [1]. This is preventing the kernel
> build from
>  generating its own .scmversion.
> 
>  If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and
> applying any
>  patch in the linux-yocto recipe, this will result in a
> non-reproducible build.
> >>
> >> ...
> >>
>  diff --git a/tools/kgit-s2q b/tools/kgit-s2q
>  index 706783e..b46a138 100755
>  --- a/tools/kgit-s2q
>  +++ b/tools/kgit-s2q
>  @@ -558,7 +558,7 @@ do
>   echo "($APPLIED/$COUNT) `basename $i`"
>   fi
> 
>  -   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
>  +   git am --keep-non-patch --committer-date-is-author-date
> $DIR/$i > /dev/null 2>&1
>   if [ $? != 0 ];then
>   git am --abort > /dev/null 2>&1
>   echo "[INFO]: check of $DIR/$i with \"git
> am\" did not pass, trying reduced context."
> 
> 
> 
>  I'm not sure if this is a proper solution to fix the problem or
> what side effects it may introduce?
> >>>
> >>> There's nothing fundamentally wrong with that solution, but there
> are
> >>> modes for the kernel builds where we absolutely do not want the
> >>> reproducible build (and older dates) to be in effect (see my
> comments
> >>> on the OE core mailing list when the ability to disable
> reproducible
> >>> builds was almost removed).
> >>
> >> Thanks, now I've looked it up.
> >>
> >>> We could add an option to the patch queue management that was
> enabled
> >>> when reproducible builds are in play, so git am could operate like
> >>> that.
> >>>
> >>> Alternatively, you could bbappend the other kernel's do_configure
> and
> >>> do the same .scmversion trick to make sure that everything is
> >>> consistent.
> >>
> >> I don't follow, can you be more specific?
> >> We encountered this problem when building linux-imx (from
> meta-freescale).
> >> Preferably the .scmversion workaround in [1] should be dropped, so
> that
> >> there is no need to reinvent/mimic the output of
> >> linux/scripts/setlocalversion, in a bitbake recipe.
> >
> > I see no compelling reason to drop the .scmversion in the main
> kernel.bbclass
> > (at least not in the short term). It has been there for a
> significant amount
> > of time, and removing it now would very likely cause some ripple
> effects
> > througout the ecosystem (it is there for a few different issues).
> While not
> > elegant, it is an available and workable solution to the problem it
> is
> > solving.
> 
>  I agree. And obviously this is not a big issue for others.
>  There have been at least two failed attempts on removing this
> [4][5][6].
> 
> > I was suggesting that anyone can bbappend any kernel recipe to do
> the similar
> > .scmversion set (assuming they aren't using the core kernel bbclass
> > configure()).
> >
> > But from your next statement, that isn't going to work as the sole
> solution,
> > since yes, that will only restore where we keep the git rev out of
> the
> > localversion.
> >
> >>
> >> To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
> >> with a reproducible output.
> >
> > And that's actually part of the reason the .scmversion is set, the
> > setlocalversion script was appending values in some configurations
> and
> > combinations of patching, etc, such that issues were popping up with
> version
> > matching.
> >
> > Part of getting that to work, definitely could be to apply patches
> with the
> > author date, but that has to be triggered on reproducible builds
> being
> > enabled (which is of course the default) and also preserving the use
> of the
> 

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Jimmy Assarsson]

On 2022-03-09 16:47, Bruce Ashfield wrote:

On Sat, Feb 26, 2022 at 2:17 AM Jimmy Assarsson
 wrote:

On 2022-02-24 16:41, Bruce Ashfield wrote:

On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson
 wrote:

On 2022-02-21 05:13, Bruce Ashfield wrote:

on 19/02/2022 Jimmy Assarsson wrote:

On 2022-02-19 20:42, Bruce Ashfield wrote:

On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
 wrote:


When patching a kernel git repository, in a recipe inheriting 
kernel-yocto.bbclass,
the resulting commit hash will become different every time the source is 
unpacked and patched.

This is a problem that causes non-reproducible builds, when the commit hash is 
built
into the kernel (CONFIG_LOCALVERSION_AUTO=y).


Currently it is not a problem in linux-yocto, since an empty .scmversion is
created in kernel_do_configure [1]. This is preventing the kernel build from
generating its own .scmversion.

If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying any
patch in the linux-yocto recipe, this will result in a non-reproducible build.


...


diff --git a/tools/kgit-s2q b/tools/kgit-s2q
index 706783e..b46a138 100755
--- a/tools/kgit-s2q
+++ b/tools/kgit-s2q
@@ -558,7 +558,7 @@ do
 echo "($APPLIED/$COUNT) `basename $i`"
 fi

-   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
+   git am --keep-non-patch --committer-date-is-author-date $DIR/$i > /dev/null 
2>&1
 if [ $? != 0 ];then
 git am --abort > /dev/null 2>&1
 echo "[INFO]: check of $DIR/$i with \"git am\" did not pass, 
trying reduced context."



I'm not sure if this is a proper solution to fix the problem or what side 
effects it may introduce?


There's nothing fundamentally wrong with that solution, but there are
modes for the kernel builds where we absolutely do not want the
reproducible build (and older dates) to be in effect (see my comments
on the OE core mailing list when the ability to disable reproducible
builds was almost removed).


Thanks, now I've looked it up.


We could add an option to the patch queue management that was enabled
when reproducible builds are in play, so git am could operate like
that.

Alternatively, you could bbappend the other kernel's do_configure and
do the same .scmversion trick to make sure that everything is
consistent.


I don't follow, can you be more specific?
We encountered this problem when building linux-imx (from meta-freescale).
Preferably the .scmversion workaround in [1] should be dropped, so that
there is no need to reinvent/mimic the output of
linux/scripts/setlocalversion, in a bitbake recipe.


I see no compelling reason to drop the .scmversion in the main kernel.bbclass
(at least not in the short term). It has been there for a significant amount
of time, and removing it now would very likely cause some ripple effects
througout the ecosystem (it is there for a few different issues).  While not
elegant, it is an available and workable solution to the problem it is
solving.


I agree. And obviously this is not a big issue for others.
There have been at least two failed attempts on removing this [4][5][6].


I was suggesting that anyone can bbappend any kernel recipe to do the similar
.scmversion set (assuming they aren't using the core kernel bbclass
configure()).

But from your next statement, that isn't going to work as the sole solution,
since yes, that will only restore where we keep the git rev out of the
localversion.



To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
with a reproducible output.


And that's actually part of the reason the .scmversion is set, the
setlocalversion script was appending values in some configurations and
combinations of patching, etc, such that issues were popping up with version
matching.

Part of getting that to work, definitely could be to apply patches with the
author date, but that has to be triggered on reproducible builds being
enabled (which is of course the default) and also preserving the use of the
.scmversion for the baseline / basic kernel configuration routines.

.. so one config to say "don't set scmversion" and another to say "apply
patches with author date" (this one toggling based on reproducible builds).
And then both scenarios can be supported.


Currently I'm only interested in fixing the "apply patches with
author date", to get reproducible builds.

So we will end up with a new config in kernel-yocto.bbclass, since this
is the only location where kgit-s2q is used.
And we will get a new option in kgit-s2q, that will result in
"git am --committer-date-is-author-date" being used.

Any name suggestions for the config and option?
KERNEL_PATCH_WITH_AUTHOR_DATE
KERNEL_PATCH_REPRODUCIBLE_HASH
KERNEL_PATCH_REPRODUCIBLE_COMMIT


It is better to abstract it behind the reproducible feature
description, since there are other kernel reproducibility components
that exist, or may appear in the future.


Sure.


If you want, I can have a go at the patch, 

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Bruce Ashfield]

On Sat, Feb 26, 2022 at 2:17 AM Jimmy Assarsson
 wrote:
>
> On 2022-02-24 16:41, Bruce Ashfield wrote:
> > On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson
> >  wrote:
> >> On 2022-02-21 05:13, Bruce Ashfield wrote:
> >>> on 19/02/2022 Jimmy Assarsson wrote:
>  On 2022-02-19 20:42, Bruce Ashfield wrote:
> > On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
> >  wrote:
> >>
> >> When patching a kernel git repository, in a recipe inheriting 
> >> kernel-yocto.bbclass,
> >> the resulting commit hash will become different every time the source 
> >> is unpacked and patched.
> >>
> >> This is a problem that causes non-reproducible builds, when the commit 
> >> hash is built
> >> into the kernel (CONFIG_LOCALVERSION_AUTO=y).
> >>
> >>
> >> Currently it is not a problem in linux-yocto, since an empty 
> >> .scmversion is
> >> created in kernel_do_configure [1]. This is preventing the kernel 
> >> build from
> >> generating its own .scmversion.
> >>
> >> If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and 
> >> applying any
> >> patch in the linux-yocto recipe, this will result in a 
> >> non-reproducible build.
> 
>  ...
> 
> >> diff --git a/tools/kgit-s2q b/tools/kgit-s2q
> >> index 706783e..b46a138 100755
> >> --- a/tools/kgit-s2q
> >> +++ b/tools/kgit-s2q
> >> @@ -558,7 +558,7 @@ do
> >> echo "($APPLIED/$COUNT) `basename $i`"
> >> fi
> >>
> >> -   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
> >> +   git am --keep-non-patch --committer-date-is-author-date 
> >> $DIR/$i > /dev/null 2>&1
> >> if [ $? != 0 ];then
> >> git am --abort > /dev/null 2>&1
> >> echo "[INFO]: check of $DIR/$i with \"git am\" did 
> >> not pass, trying reduced context."
> >>
> >>
> >>
> >> I'm not sure if this is a proper solution to fix the problem or what 
> >> side effects it may introduce?
> >
> > There's nothing fundamentally wrong with that solution, but there are
> > modes for the kernel builds where we absolutely do not want the
> > reproducible build (and older dates) to be in effect (see my comments
> > on the OE core mailing list when the ability to disable reproducible
> > builds was almost removed).
> 
>  Thanks, now I've looked it up.
> 
> > We could add an option to the patch queue management that was enabled
> > when reproducible builds are in play, so git am could operate like
> > that.
> >
> > Alternatively, you could bbappend the other kernel's do_configure and
> > do the same .scmversion trick to make sure that everything is
> > consistent.
> 
>  I don't follow, can you be more specific?
>  We encountered this problem when building linux-imx (from 
>  meta-freescale).
>  Preferably the .scmversion workaround in [1] should be dropped, so that
>  there is no need to reinvent/mimic the output of
>  linux/scripts/setlocalversion, in a bitbake recipe.
> >>>
> >>> I see no compelling reason to drop the .scmversion in the main 
> >>> kernel.bbclass
> >>> (at least not in the short term). It has been there for a significant 
> >>> amount
> >>> of time, and removing it now would very likely cause some ripple effects
> >>> througout the ecosystem (it is there for a few different issues).  While 
> >>> not
> >>> elegant, it is an available and workable solution to the problem it is
> >>> solving.
> >>
> >> I agree. And obviously this is not a big issue for others.
> >> There have been at least two failed attempts on removing this [4][5][6].
> >>
> >>> I was suggesting that anyone can bbappend any kernel recipe to do the 
> >>> similar
> >>> .scmversion set (assuming they aren't using the core kernel bbclass
> >>> configure()).
> >>>
> >>> But from your next statement, that isn't going to work as the sole 
> >>> solution,
> >>> since yes, that will only restore where we keep the git rev out of the
> >>> localversion.
> >>>
> 
>  To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
>  with a reproducible output.
> >>>
> >>> And that's actually part of the reason the .scmversion is set, the
> >>> setlocalversion script was appending values in some configurations and
> >>> combinations of patching, etc, such that issues were popping up with 
> >>> version
> >>> matching.
> >>>
> >>> Part of getting that to work, definitely could be to apply patches with 
> >>> the
> >>> author date, but that has to be triggered on reproducible builds being
> >>> enabled (which is of course the default) and also preserving the use of 
> >>> the
> >>> .scmversion for the baseline / basic kernel configuration routines.
> >>>
> >>> .. so one config to say "don't set scmversion" and another to say "apply
> >>> patches with author date" (this one 

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Jimmy Assarsson]

On 2022-02-24 16:41, Bruce Ashfield wrote:

On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson
 wrote:

On 2022-02-21 05:13, Bruce Ashfield wrote:

on 19/02/2022 Jimmy Assarsson wrote:

On 2022-02-19 20:42, Bruce Ashfield wrote:

On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
 wrote:


When patching a kernel git repository, in a recipe inheriting 
kernel-yocto.bbclass,
the resulting commit hash will become different every time the source is 
unpacked and patched.

This is a problem that causes non-reproducible builds, when the commit hash is 
built
into the kernel (CONFIG_LOCALVERSION_AUTO=y).


Currently it is not a problem in linux-yocto, since an empty .scmversion is
created in kernel_do_configure [1]. This is preventing the kernel build from
generating its own .scmversion.

If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying any
patch in the linux-yocto recipe, this will result in a non-reproducible build.


...


diff --git a/tools/kgit-s2q b/tools/kgit-s2q
index 706783e..b46a138 100755
--- a/tools/kgit-s2q
+++ b/tools/kgit-s2q
@@ -558,7 +558,7 @@ do
echo "($APPLIED/$COUNT) `basename $i`"
fi

-   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
+   git am --keep-non-patch --committer-date-is-author-date $DIR/$i > /dev/null 
2>&1
if [ $? != 0 ];then
git am --abort > /dev/null 2>&1
echo "[INFO]: check of $DIR/$i with \"git am\" did not pass, 
trying reduced context."



I'm not sure if this is a proper solution to fix the problem or what side 
effects it may introduce?


There's nothing fundamentally wrong with that solution, but there are
modes for the kernel builds where we absolutely do not want the
reproducible build (and older dates) to be in effect (see my comments
on the OE core mailing list when the ability to disable reproducible
builds was almost removed).


Thanks, now I've looked it up.


We could add an option to the patch queue management that was enabled
when reproducible builds are in play, so git am could operate like
that.

Alternatively, you could bbappend the other kernel's do_configure and
do the same .scmversion trick to make sure that everything is
consistent.


I don't follow, can you be more specific?
We encountered this problem when building linux-imx (from meta-freescale).
Preferably the .scmversion workaround in [1] should be dropped, so that
there is no need to reinvent/mimic the output of
linux/scripts/setlocalversion, in a bitbake recipe.


I see no compelling reason to drop the .scmversion in the main kernel.bbclass
(at least not in the short term). It has been there for a significant amount
of time, and removing it now would very likely cause some ripple effects
througout the ecosystem (it is there for a few different issues).  While not
elegant, it is an available and workable solution to the problem it is
solving.


I agree. And obviously this is not a big issue for others.
There have been at least two failed attempts on removing this [4][5][6].


I was suggesting that anyone can bbappend any kernel recipe to do the similar
.scmversion set (assuming they aren't using the core kernel bbclass
configure()).

But from your next statement, that isn't going to work as the sole solution,
since yes, that will only restore where we keep the git rev out of the
localversion.



To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
with a reproducible output.


And that's actually part of the reason the .scmversion is set, the
setlocalversion script was appending values in some configurations and
combinations of patching, etc, such that issues were popping up with version
matching.

Part of getting that to work, definitely could be to apply patches with the
author date, but that has to be triggered on reproducible builds being
enabled (which is of course the default) and also preserving the use of the
.scmversion for the baseline / basic kernel configuration routines.

.. so one config to say "don't set scmversion" and another to say "apply
patches with author date" (this one toggling based on reproducible builds).
And then both scenarios can be supported.


Currently I'm only interested in fixing the "apply patches with
author date", to get reproducible builds.

So we will end up with a new config in kernel-yocto.bbclass, since this
is the only location where kgit-s2q is used.
And we will get a new option in kgit-s2q, that will result in
"git am --committer-date-is-author-date" being used.

Any name suggestions for the config and option?
KERNEL_PATCH_WITH_AUTHOR_DATE
KERNEL_PATCH_REPRODUCIBLE_HASH
KERNEL_PATCH_REPRODUCIBLE_COMMIT


It is better to abstract it behind the reproducible feature
description, since there are other kernel reproducibility components
that exist, or may appear in the future.


Sure.


If you want, I can have a go at the patch, which will allow me to run
my local tests and put it in my next consolidated pull request.


That would be 

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Bruce Ashfield]

On Tue, Feb 22, 2022 at 3:42 AM Jimmy Assarsson
 wrote:
>
> On 2022-02-21 05:13, Bruce Ashfield wrote:
> > on 19/02/2022 Jimmy Assarsson wrote:
> >> On 2022-02-19 20:42, Bruce Ashfield wrote:
> >>> On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
> >>>  wrote:
> 
>  When patching a kernel git repository, in a recipe inheriting 
>  kernel-yocto.bbclass,
>  the resulting commit hash will become different every time the source is 
>  unpacked and patched.
> 
>  This is a problem that causes non-reproducible builds, when the commit 
>  hash is built
>  into the kernel (CONFIG_LOCALVERSION_AUTO=y).
> 
> 
>  Currently it is not a problem in linux-yocto, since an empty .scmversion 
>  is
>  created in kernel_do_configure [1]. This is preventing the kernel build 
>  from
>  generating its own .scmversion.
> 
>  If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying 
>  any
>  patch in the linux-yocto recipe, this will result in a non-reproducible 
>  build.
> >>
> >> ...
> >>
>  diff --git a/tools/kgit-s2q b/tools/kgit-s2q
>  index 706783e..b46a138 100755
>  --- a/tools/kgit-s2q
>  +++ b/tools/kgit-s2q
>  @@ -558,7 +558,7 @@ do
> echo "($APPLIED/$COUNT) `basename $i`"
> fi
> 
>  -   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
>  +   git am --keep-non-patch --committer-date-is-author-date $DIR/$i 
>  > /dev/null 2>&1
> if [ $? != 0 ];then
> git am --abort > /dev/null 2>&1
> echo "[INFO]: check of $DIR/$i with \"git am\" did 
>  not pass, trying reduced context."
> 
> 
> 
>  I'm not sure if this is a proper solution to fix the problem or what 
>  side effects it may introduce?
> >>>
> >>> There's nothing fundamentally wrong with that solution, but there are
> >>> modes for the kernel builds where we absolutely do not want the
> >>> reproducible build (and older dates) to be in effect (see my comments
> >>> on the OE core mailing list when the ability to disable reproducible
> >>> builds was almost removed).
> >>
> >> Thanks, now I've looked it up.
> >>
> >>> We could add an option to the patch queue management that was enabled
> >>> when reproducible builds are in play, so git am could operate like
> >>> that.
> >>>
> >>> Alternatively, you could bbappend the other kernel's do_configure and
> >>> do the same .scmversion trick to make sure that everything is
> >>> consistent.
> >>
> >> I don't follow, can you be more specific?
> >> We encountered this problem when building linux-imx (from meta-freescale).
> >> Preferably the .scmversion workaround in [1] should be dropped, so that
> >> there is no need to reinvent/mimic the output of
> >> linux/scripts/setlocalversion, in a bitbake recipe.
> >
> > I see no compelling reason to drop the .scmversion in the main 
> > kernel.bbclass
> > (at least not in the short term). It has been there for a significant amount
> > of time, and removing it now would very likely cause some ripple effects
> > througout the ecosystem (it is there for a few different issues).  While not
> > elegant, it is an available and workable solution to the problem it is
> > solving.
>
> I agree. And obviously this is not a big issue for others.
> There have been at least two failed attempts on removing this [4][5][6].
>
> > I was suggesting that anyone can bbappend any kernel recipe to do the 
> > similar
> > .scmversion set (assuming they aren't using the core kernel bbclass
> > configure()).
> >
> > But from your next statement, that isn't going to work as the sole solution,
> > since yes, that will only restore where we keep the git rev out of the
> > localversion.
> >
> >>
> >> To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
> >> with a reproducible output.
> >
> > And that's actually part of the reason the .scmversion is set, the
> > setlocalversion script was appending values in some configurations and
> > combinations of patching, etc, such that issues were popping up with version
> > matching.
> >
> > Part of getting that to work, definitely could be to apply patches with the
> > author date, but that has to be triggered on reproducible builds being
> > enabled (which is of course the default) and also preserving the use of the
> > .scmversion for the baseline / basic kernel configuration routines.
> >
> > .. so one config to say "don't set scmversion" and another to say "apply
> > patches with author date" (this one toggling based on reproducible builds).
> > And then both scenarios can be supported.
>
> Currently I'm only interested in fixing the "apply patches with
> author date", to get reproducible builds.
>
> So we will end up with a new config in kernel-yocto.bbclass, since this
> is the only location where kgit-s2q is used.
> And we will get a new option in kgit-s2q, that will result in
> 

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Jimmy Assarsson]

On 2022-02-21 05:13, Bruce Ashfield wrote:

on 19/02/2022 Jimmy Assarsson wrote:

On 2022-02-19 20:42, Bruce Ashfield wrote:

On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
 wrote:


When patching a kernel git repository, in a recipe inheriting 
kernel-yocto.bbclass,
the resulting commit hash will become different every time the source is 
unpacked and patched.

This is a problem that causes non-reproducible builds, when the commit hash is 
built
into the kernel (CONFIG_LOCALVERSION_AUTO=y).


Currently it is not a problem in linux-yocto, since an empty .scmversion is
created in kernel_do_configure [1]. This is preventing the kernel build from
generating its own .scmversion.

If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying any
patch in the linux-yocto recipe, this will result in a non-reproducible build.


...


diff --git a/tools/kgit-s2q b/tools/kgit-s2q
index 706783e..b46a138 100755
--- a/tools/kgit-s2q
+++ b/tools/kgit-s2q
@@ -558,7 +558,7 @@ do
   echo "($APPLIED/$COUNT) `basename $i`"
   fi

-   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
+   git am --keep-non-patch --committer-date-is-author-date $DIR/$i > /dev/null 
2>&1
   if [ $? != 0 ];then
   git am --abort > /dev/null 2>&1
   echo "[INFO]: check of $DIR/$i with \"git am\" did not pass, 
trying reduced context."



I'm not sure if this is a proper solution to fix the problem or what side 
effects it may introduce?


There's nothing fundamentally wrong with that solution, but there are
modes for the kernel builds where we absolutely do not want the
reproducible build (and older dates) to be in effect (see my comments
on the OE core mailing list when the ability to disable reproducible
builds was almost removed).


Thanks, now I've looked it up.


We could add an option to the patch queue management that was enabled
when reproducible builds are in play, so git am could operate like
that.

Alternatively, you could bbappend the other kernel's do_configure and
do the same .scmversion trick to make sure that everything is
consistent.


I don't follow, can you be more specific?
We encountered this problem when building linux-imx (from meta-freescale).
Preferably the .scmversion workaround in [1] should be dropped, so that
there is no need to reinvent/mimic the output of
linux/scripts/setlocalversion, in a bitbake recipe.


I see no compelling reason to drop the .scmversion in the main kernel.bbclass
(at least not in the short term). It has been there for a significant amount
of time, and removing it now would very likely cause some ripple effects
througout the ecosystem (it is there for a few different issues).  While not
elegant, it is an available and workable solution to the problem it is
solving.


I agree. And obviously this is not a big issue for others.
There have been at least two failed attempts on removing this [4][5][6].


I was suggesting that anyone can bbappend any kernel recipe to do the similar
.scmversion set (assuming they aren't using the core kernel bbclass
configure()).

But from your next statement, that isn't going to work as the sole solution,
since yes, that will only restore where we keep the git rev out of the
localversion.



To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
with a reproducible output.


And that's actually part of the reason the .scmversion is set, the
setlocalversion script was appending values in some configurations and
combinations of patching, etc, such that issues were popping up with version
matching.

Part of getting that to work, definitely could be to apply patches with the
author date, but that has to be triggered on reproducible builds being
enabled (which is of course the default) and also preserving the use of the
.scmversion for the baseline / basic kernel configuration routines.

.. so one config to say "don't set scmversion" and another to say "apply
patches with author date" (this one toggling based on reproducible builds).
And then both scenarios can be supported.


Currently I'm only interested in fixing the "apply patches with
author date", to get reproducible builds.

So we will end up with a new config in kernel-yocto.bbclass, since this
is the only location where kgit-s2q is used.
And we will get a new option in kgit-s2q, that will result in
"git am --committer-date-is-author-date" being used.

Any name suggestions for the config and option?
KERNEL_PATCH_WITH_AUTHOR_DATE
KERNEL_PATCH_REPRODUCIBLE_HASH
KERNEL_PATCH_REPRODUCIBLE_COMMIT
--patch-with-author-date
--apply-patch-with-author-date
--committer-date-is-author-date


[4] https://patchwork.openembedded.org/patch/138333
[5] https://patchwork.openembedded.org/patch/171479
[6] https://patchwork.openembedded.org/patch/171513


Best regards,
jimmy



Bruce



Best regards,
jimmy



Bruce



[1] 
https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589
[2] 

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Bruce Ashfield]

In message: Re: Non-deterministic SHA commit hash when applying patches to 
kernel git repository
on 19/02/2022 Jimmy Assarsson wrote:

> On 2022-02-19 20:42, Bruce Ashfield wrote:
> > On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
> >  wrote:
> > > 
> > > When patching a kernel git repository, in a recipe inheriting 
> > > kernel-yocto.bbclass,
> > > the resulting commit hash will become different every time the source is 
> > > unpacked and patched.
> > > 
> > > This is a problem that causes non-reproducible builds, when the commit 
> > > hash is built
> > > into the kernel (CONFIG_LOCALVERSION_AUTO=y).
> > > 
> > > 
> > > Currently it is not a problem in linux-yocto, since an empty .scmversion 
> > > is
> > > created in kernel_do_configure [1]. This is preventing the kernel build 
> > > from
> > > generating its own .scmversion.
> > > 
> > > If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying 
> > > any
> > > patch in the linux-yocto recipe, this will result in a non-reproducible 
> > > build.
> 
> ...
> 
> > > diff --git a/tools/kgit-s2q b/tools/kgit-s2q
> > > index 706783e..b46a138 100755
> > > --- a/tools/kgit-s2q
> > > +++ b/tools/kgit-s2q
> > > @@ -558,7 +558,7 @@ do
> > >   echo "($APPLIED/$COUNT) `basename $i`"
> > >   fi
> > > 
> > > -   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
> > > +   git am --keep-non-patch --committer-date-is-author-date $DIR/$i > 
> > > /dev/null 2>&1
> > >   if [ $? != 0 ];then
> > >   git am --abort > /dev/null 2>&1
> > >   echo "[INFO]: check of $DIR/$i with \"git am\" did not 
> > > pass, trying reduced context."
> > > 
> > > 
> > > 
> > > I'm not sure if this is a proper solution to fix the problem or what side 
> > > effects it may introduce?
> > 
> > There's nothing fundamentally wrong with that solution, but there are
> > modes for the kernel builds where we absolutely do not want the
> > reproducible build (and older dates) to be in effect (see my comments
> > on the OE core mailing list when the ability to disable reproducible
> > builds was almost removed).
> 
> Thanks, now I've looked it up.
> 
> > We could add an option to the patch queue management that was enabled
> > when reproducible builds are in play, so git am could operate like
> > that.
> > 
> > Alternatively, you could bbappend the other kernel's do_configure and
> > do the same .scmversion trick to make sure that everything is
> > consistent.
> 
> I don't follow, can you be more specific?
> We encountered this problem when building linux-imx (from meta-freescale).
> Preferably the .scmversion workaround in [1] should be dropped, so that
> there is no need to reinvent/mimic the output of
> linux/scripts/setlocalversion, in a bitbake recipe.

I see no compelling reason to drop the .scmversion in the main kernel.bbclass
(at least not in the short term). It has been there for a significant amount
of time, and removing it now would very likely cause some ripple effects
througout the ecosystem (it is there for a few different issues).  While not
elegant, it is an available and workable solution to the problem it is
solving.

I was suggesting that anyone can bbappend any kernel recipe to do the similar
.scmversion set (assuming they aren't using the core kernel bbclass
configure()).

But from your next statement, that isn't going to work as the sole solution,
since yes, that will only restore where we keep the git rev out of the
localversion.

> 
> To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
> with a reproducible output.

And that's actually part of the reason the .scmversion is set, the
setlocalversion script was appending values in some configurations and
combinations of patching, etc, such that issues were popping up with version
matching.

Part of getting that to work, definitely could be to apply patches with the
author date, but that has to be triggered on reproducible builds being
enabled (which is of course the default) and also preserving the use of the
.scmversion for the baseline / basic kernel configuration routines.

.. so one config to say "don't set scmversion" and another to say "apply
patches with author date" (this one toggling based on reproducible builds).
And then both scenarios can be supported.

Bruce

> 
> Best regards,
> jimmy
> 
> 
> > Bruce
> > 
> > > 
> > > [1] 
> > > https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589
> > > [2] 
> > > https://git.yoctoproject.org/meta-freescale/tree/classes/fsl-kernel-localversion.bbclass?h=honister
> > > [3] 
> > > https://github.com/OE4T/meta-tegra/blob/honister/recipes-kernel/linux/linux-tegra_4.9.bb

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10970): 
https://lists.yoctoproject.org/g/linux-yocto/message/10970
Mute This Topic: https://lists.yoctoproject.org/mt/89251386/21656
Group Owner: linux-yocto+ow...@lists.yoctoproject.org

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Jimmy Assarsson]

On 2022-02-19 20:42, Bruce Ashfield wrote:

On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
 wrote:


When patching a kernel git repository, in a recipe inheriting 
kernel-yocto.bbclass,
the resulting commit hash will become different every time the source is 
unpacked and patched.

This is a problem that causes non-reproducible builds, when the commit hash is 
built
into the kernel (CONFIG_LOCALVERSION_AUTO=y).


Currently it is not a problem in linux-yocto, since an empty .scmversion is
created in kernel_do_configure [1]. This is preventing the kernel build from
generating its own .scmversion.

If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying any
patch in the linux-yocto recipe, this will result in a non-reproducible build.


...


diff --git a/tools/kgit-s2q b/tools/kgit-s2q
index 706783e..b46a138 100755
--- a/tools/kgit-s2q
+++ b/tools/kgit-s2q
@@ -558,7 +558,7 @@ do
  echo "($APPLIED/$COUNT) `basename $i`"
  fi

-   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
+   git am --keep-non-patch --committer-date-is-author-date $DIR/$i > /dev/null 
2>&1
  if [ $? != 0 ];then
  git am --abort > /dev/null 2>&1
  echo "[INFO]: check of $DIR/$i with \"git am\" did not pass, 
trying reduced context."



I'm not sure if this is a proper solution to fix the problem or what side 
effects it may introduce?


There's nothing fundamentally wrong with that solution, but there are
modes for the kernel builds where we absolutely do not want the
reproducible build (and older dates) to be in effect (see my comments
on the OE core mailing list when the ability to disable reproducible
builds was almost removed).


Thanks, now I've looked it up.


We could add an option to the patch queue management that was enabled
when reproducible builds are in play, so git am could operate like
that.

Alternatively, you could bbappend the other kernel's do_configure and
do the same .scmversion trick to make sure that everything is
consistent.


I don't follow, can you be more specific?
We encountered this problem when building linux-imx (from meta-freescale).
Preferably the .scmversion workaround in [1] should be dropped, so that
there is no need to reinvent/mimic the output of
linux/scripts/setlocalversion, in a bitbake recipe.

To be clear, we want the result of CONFIG_LOCALVERSION_AUTO=y, but
with a reproducible output.

Best regards,
jimmy



Bruce



[1] 
https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589
[2] 
https://git.yoctoproject.org/meta-freescale/tree/classes/fsl-kernel-localversion.bbclass?h=honister
[3] 
https://github.com/OE4T/meta-tegra/blob/honister/recipes-kernel/linux/linux-tegra_4.9.bb

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10966): 
https://lists.yoctoproject.org/g/linux-yocto/message/10966
Mute This Topic: https://lists.yoctoproject.org/mt/89251386/21656
Group Owner: linux-yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Bruce Ashfield]

On Sat, Feb 19, 2022 at 5:28 AM Jimmy Assarsson
 wrote:
>
> When patching a kernel git repository, in a recipe inheriting 
> kernel-yocto.bbclass,
> the resulting commit hash will become different every time the source is 
> unpacked and patched.
>
> This is a problem that causes non-reproducible builds, when the commit hash 
> is built
> into the kernel (CONFIG_LOCALVERSION_AUTO=y).
>
>
> Currently it is not a problem in linux-yocto, since an empty .scmversion is
> created in kernel_do_configure [1]. This is preventing the kernel build from
> generating its own .scmversion.
>
> If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying any
> patch in the linux-yocto recipe, this will result in a non-reproducible build.
>
>
> However, there are other kernel recipes where this is already a problem;
> meta-freescale [2] and meta-tegra [3].
>
> Currently we solve this problem by using the --committer-date-is-author-date
> "git am" option, when applying patches with kgit-s2q:
>  From f2b6cdb745e7a7965497efccff415ec81207f4d7 Mon Sep 17 00:00:00 2001
> From: Jimmy Assarsson 
> Date: Fri, 4 Feb 2022 17:58:29 +0100
> Subject: [PATCH] kgit-s2q: call "git am" with --committer-date-is-author-date
>
> Produce deterministic commit hashes, by passing the
> "--committer-date-is-author-date" option when applying patches with "git am".
>
> Signed-off-by: Jimmy Assarsson 
> ---
>   tools/kgit-s2q | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tools/kgit-s2q b/tools/kgit-s2q
> index 706783e..b46a138 100755
> --- a/tools/kgit-s2q
> +++ b/tools/kgit-s2q
> @@ -558,7 +558,7 @@ do
>  echo "($APPLIED/$COUNT) `basename $i`"
>  fi
>
> -   git am --keep-non-patch $DIR/$i > /dev/null 2>&1
> +   git am --keep-non-patch --committer-date-is-author-date $DIR/$i > 
> /dev/null 2>&1
>  if [ $? != 0 ];then
>  git am --abort > /dev/null 2>&1
>  echo "[INFO]: check of $DIR/$i with \"git am\" did not pass, 
> trying reduced context."
>
>
>
> I'm not sure if this is a proper solution to fix the problem or what side 
> effects it may introduce?

There's nothing fundamentally wrong with that solution, but there are
modes for the kernel builds where we absolutely do not want the
reproducible build (and older dates) to be in effect (see my comments
on the OE core mailing list when the ability to disable reproducible
builds was almost removed).

We could add an option to the patch queue management that was enabled
when reproducible builds are in play, so git am could operate like
that.

Alternatively, you could bbappend the other kernel's do_configure and
do the same .scmversion trick to make sure that everything is
consistent.

Bruce

>
> [1] 
> https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589
> [2] 
> https://git.yoctoproject.org/meta-freescale/tree/classes/fsl-kernel-localversion.bbclass?h=honister
> [3] 
> https://github.com/OE4T/meta-tegra/blob/honister/recipes-kernel/linux/linux-tegra_4.9.bb
>
> Best regards,
> jimmy



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10964): 
https://lists.yoctoproject.org/g/linux-yocto/message/10964
Mute This Topic: https://lists.yoctoproject.org/mt/89251386/21656
Group Owner: linux-yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[linux-yocto] Non-deterministic SHA commit hash when applying patches to kernel git repository

[Jimmy Assarsson]

When patching a kernel git repository, in a recipe inheriting 
kernel-yocto.bbclass,
the resulting commit hash will become different every time the source is 
unpacked and patched.

This is a problem that causes non-reproducible builds, when the commit hash is 
built
into the kernel (CONFIG_LOCALVERSION_AUTO=y).


Currently it is not a problem in linux-yocto, since an empty .scmversion is
created in kernel_do_configure [1]. This is preventing the kernel build from
generating its own .scmversion.

If removing this commit, setting CONFIG_LOCALVERSION_AUTO=y and applying any
patch in the linux-yocto recipe, this will result in a non-reproducible build.


However, there are other kernel recipes where this is already a problem;
meta-freescale [2] and meta-tegra [3].

Currently we solve this problem by using the --committer-date-is-author-date
"git am" option, when applying patches with kgit-s2q:
From f2b6cdb745e7a7965497efccff415ec81207f4d7 Mon Sep 17 00:00:00 2001
From: Jimmy Assarsson 
Date: Fri, 4 Feb 2022 17:58:29 +0100
Subject: [PATCH] kgit-s2q: call "git am" with --committer-date-is-author-date

Produce deterministic commit hashes, by passing the
"--committer-date-is-author-date" option when applying patches with "git am".

Signed-off-by: Jimmy Assarsson 
---
 tools/kgit-s2q | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/kgit-s2q b/tools/kgit-s2q
index 706783e..b46a138 100755
--- a/tools/kgit-s2q
+++ b/tools/kgit-s2q
@@ -558,7 +558,7 @@ do
echo "($APPLIED/$COUNT) `basename $i`"
fi
 
-   git am --keep-non-patch $DIR/$i > /dev/null 2>&1

+   git am --keep-non-patch --committer-date-is-author-date $DIR/$i > /dev/null 
2>&1
if [ $? != 0 ];then
git am --abort > /dev/null 2>&1
echo "[INFO]: check of $DIR/$i with \"git am\" did not pass, trying 
reduced context."



I'm not sure if this is a proper solution to fix the problem or what side 
effects it may introduce?

[1] 
https://git.yoctoproject.org/poky/tree/meta/classes/kernel.bbclass?h=honister#n589
[2] 
https://git.yoctoproject.org/meta-freescale/tree/classes/fsl-kernel-localversion.bbclass?h=honister
[3] 
https://github.com/OE4T/meta-tegra/blob/honister/recipes-kernel/linux/linux-tegra_4.9.bb

Best regards,
jimmy

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10963): 
https://lists.yoctoproject.org/g/linux-yocto/message/10963
Mute This Topic: https://lists.yoctoproject.org/mt/89251386/21656
Group Owner: linux-yocto+ow...@lists.yoctoproject.org
Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-