Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Fri, May 5, 2023 at 6:24 AM Jose Quaresma  wrote:
>
> Hi Bruce,
>
> Jose Quaresma via lists.openembedded.org 
>  escreveu no dia quarta, 
> 3/05/2023 à(s) 11:09:
>>
>>
>>
>> Bruce Ashfield  escreveu no dia terça, 2/05/2023 
>> à(s) 22:12:
>>>
>>> Attached is v2 of the patch. I've consolidated the suggested changes.
>>>
>>> I'm soaking it a bit longer, and then will send it as part of my next
>>> consolidated pull request.
>>
>>
>> I will do some more tests with the v2 and post my comment later if anything 
>> new comes up.
>
>
> Nothing new and the v2 patch works pretty well in my kernels.
>

Awesome. Thanks for the test and update!

Bruce

> Jose
>
>>
>>
>> Jose
>>
>>>
>>>
>>> Bruce
>>>
>>> On Thu, Apr 27, 2023 at 9:26 PM Bruce Ashfield via
>>> lists.openembedded.org
>>>  wrote:
>>> >
>>> > On Thu, Apr 27, 2023 at 6:32 PM Jose Quaresma  
>>> > wrote:
>>> > >
>>> > > Hi Bruce,
>>> > >
>>> > > I have been testing your patch and have some comments.
>>> > > In some of my kernels I don't have the pkg-config changes and so I have 
>>> > > some fails linking the scripts/sign-file
>>> > > because for static linking with the libcrypto we need the -ldl -pthread.
>>> > >
>>> > > To fix my build I need to override the CRYPTO_LIBS in my kernel because 
>>> > > they use the hardcoded pkg-config
>>> > > where it is not possible to pass the --static argument.
>>> > >
>>> > > With following change on top of your patch I can build moist of my 
>>> > > kernels:
>>> > >
>>> > > export HOSTLDFLAGS="-lz"
>>> > >
>>> > > +   HOSTPKG_CONFIG="pkg-config --static"
>>> > > +   # override CRYPTO_LIBS since HOSTPKG_CONFIG lands only in 
>>> > > v5.19-rc1
>>> > > +   # 
>>> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
>>> > > +   CRYPTO_LIBS="$(pkg-config --static --libs libcrypto 2>/dev/null 
>>> > > || echo -lcrypto)"
>>> > > +
>>> > > unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
>>> > > for t in prepare scripts_basic scripts; do
>>> > > oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
>>> > > AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
>>> > > -   HOSTPKG_CONFIG="pkg-config --static" \
>>> > > +   HOSTPKG_CONFIG="${HOSTPKG_CONFIG}" 
>>> > > CRYPTO_LIBS="${CRYPTO_LIBS}" \
>>> > > -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
>>> > > done
>>> > >
>>> > >
>>> > > I think belive that the LIBELF_LIBS needs the same fix for the cases 
>>> > > where HOSTPKG_CONFIG is not available.
>>> > > Also I think there is a typo in the LIBELF_LIBS because you first 
>>> > > populate it with pkg-config but on the export the variable is redefined
>>> > > with the HOST_LIBELF_LIBS. I made a litle change too on that:
>>> > >
>>> > > # for pre-5.15 kernels
>>> > > -   LIBELF_LIBS=$(pkg-config libelf --libs 2>/dev/null || echo 
>>> > > -lelf)
>>> > > -   export LIBELF_LIBS="$HOST_LIBELF_LIBS -lz"
>>> > > +   LIBELF_LIBS=$(pkg-config --static libelf --libs 2>/dev/null || 
>>> > > echo -lelf)
>>> > > +   export LIBELF_LIBS="$LIBELF_LIBS -lz"
>>> > > export HOSTLDFLAGS="-lz"
>>> > >
>>> > > Thanks for you help.
>>> >
>>> > Those are definitely plausible tweaks to the patch, I was providing
>>> > the two techniques for that reason, and you've used them
>>> > appropriately.
>>> >
>>> > Let me roll your changes into my patch, re-test and I'll submit it to
>>> > the mailing list as a v2.
>>> >
>>> > Thanks for the testing, and fixup, I knew there would be things missing! 
>>> > :)
>>> >
>>> > Bruce
>>> >
>>> > >
>>> > > Jose
>>> > >
>>> > > Bruce Ashfield  escreveu no dia segunda, 
>>> > > 24/04/2023 à(s) 20:25:
>>> > >>
>>> > >> On Mon, Apr 24, 2023 at 6:30 AM Jose Quaresma 
>>> > >>  wrote:
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > Bruce Ashfield  escreveu no dia domingo, 
>>> > >> > 23/04/2023 à(s) 20:55:
>>> > >> >>
>>> > >> >> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
>>> > >> >>  wrote:
>>> > >> >> >
>>> > >> >> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
>>> > >> >> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
>>> > >> >> > > lists.openembedded.org
>>> > >> >> > >  wrote:
>>> > >> >> > >>
>>> > >> >> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
>>> > >> >> > >>  wrote:
>>> > >> >> > >>>
>>> > >> >> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
>>> > >> >> >  Hi,
>>> > >> >> > 
>>> > >> >> >  Not related with the previous discussion but just for
>>> > >> >> >  your information.
>>> > >> >> >  The rm_work.bbclass has an exception for the kernel recipes 
>>> > >> >> >  [1].
>>> > >> >> >  So I don't understand why we can't do the same for the 
>>> > >> >> >  make-mod-
>>> > >> >> >  scripts
>>> > >> >> >  who is the twin brother of all these kernel recipes.
>>> > >> >> > 
>>> > >> >> >  [1]
>>> > >> >> >  

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Hi Bruce,

Jose Quaresma via lists.openembedded.org  escreveu no dia quarta, 3/05/2023 à(s)
11:09:

>
>
> Bruce Ashfield  escreveu no dia terça,
> 2/05/2023 à(s) 22:12:
>
>> Attached is v2 of the patch. I've consolidated the suggested changes.
>>
>> I'm soaking it a bit longer, and then will send it as part of my next
>> consolidated pull request.
>>
>
> I will do some more tests with the v2 and post my comment later if
> anything new comes up.
>

Nothing new and the v2 patch works pretty well in my kernels.

Jose


>
> Jose
>
>
>>
>> Bruce
>>
>> On Thu, Apr 27, 2023 at 9:26 PM Bruce Ashfield via
>> lists.openembedded.org
>>  wrote:
>> >
>> > On Thu, Apr 27, 2023 at 6:32 PM Jose Quaresma 
>> wrote:
>> > >
>> > > Hi Bruce,
>> > >
>> > > I have been testing your patch and have some comments.
>> > > In some of my kernels I don't have the pkg-config changes and so I
>> have some fails linking the scripts/sign-file
>> > > because for static linking with the libcrypto we need the -ldl
>> -pthread.
>> > >
>> > > To fix my build I need to override the CRYPTO_LIBS in my kernel
>> because they use the hardcoded pkg-config
>> > > where it is not possible to pass the --static argument.
>> > >
>> > > With following change on top of your patch I can build moist of my
>> kernels:
>> > >
>> > > export HOSTLDFLAGS="-lz"
>> > >
>> > > +   HOSTPKG_CONFIG="pkg-config --static"
>> > > +   # override CRYPTO_LIBS since HOSTPKG_CONFIG lands only in
>> v5.19-rc1
>> > > +   #
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
>> > > +   CRYPTO_LIBS="$(pkg-config --static --libs libcrypto
>> 2>/dev/null || echo -lcrypto)"
>> > > +
>> > > unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
>> > > for t in prepare scripts_basic scripts; do
>> > > oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
>> > > AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
>> > > -   HOSTPKG_CONFIG="pkg-config --static" \
>> > > +   HOSTPKG_CONFIG="${HOSTPKG_CONFIG}"
>> CRYPTO_LIBS="${CRYPTO_LIBS}" \
>> > > -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR}
>> $t
>> > > done
>> > >
>> > >
>> > > I think belive that the LIBELF_LIBS needs the same fix for the cases
>> where HOSTPKG_CONFIG is not available.
>> > > Also I think there is a typo in the LIBELF_LIBS because you first
>> populate it with pkg-config but on the export the variable is redefined
>> > > with the HOST_LIBELF_LIBS. I made a litle change too on that:
>> > >
>> > > # for pre-5.15 kernels
>> > > -   LIBELF_LIBS=$(pkg-config libelf --libs 2>/dev/null || echo
>> -lelf)
>> > > -   export LIBELF_LIBS="$HOST_LIBELF_LIBS -lz"
>> > > +   LIBELF_LIBS=$(pkg-config --static libelf --libs 2>/dev/null
>> || echo -lelf)
>> > > +   export LIBELF_LIBS="$LIBELF_LIBS -lz"
>> > > export HOSTLDFLAGS="-lz"
>> > >
>> > > Thanks for you help.
>> >
>> > Those are definitely plausible tweaks to the patch, I was providing
>> > the two techniques for that reason, and you've used them
>> > appropriately.
>> >
>> > Let me roll your changes into my patch, re-test and I'll submit it to
>> > the mailing list as a v2.
>> >
>> > Thanks for the testing, and fixup, I knew there would be things
>> missing! :)
>> >
>> > Bruce
>> >
>> > >
>> > > Jose
>> > >
>> > > Bruce Ashfield  escreveu no dia segunda,
>> 24/04/2023 à(s) 20:25:
>> > >>
>> > >> On Mon, Apr 24, 2023 at 6:30 AM Jose Quaresma <
>> quaresma.j...@gmail.com> wrote:
>> > >> >
>> > >> >
>> > >> >
>> > >> > Bruce Ashfield  escreveu no dia
>> domingo, 23/04/2023 à(s) 20:55:
>> > >> >>
>> > >> >> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
>> > >> >>  wrote:
>> > >> >> >
>> > >> >> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
>> > >> >> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
>> > >> >> > > lists.openembedded.org
>> > >> >> > >  wrote:
>> > >> >> > >>
>> > >> >> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
>> > >> >> > >>  wrote:
>> > >> >> > >>>
>> > >> >> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
>> > >> >> >  Hi,
>> > >> >> > 
>> > >> >> >  Not related with the previous discussion but just for
>> > >> >> >  your information.
>> > >> >> >  The rm_work.bbclass has an exception for the kernel
>> recipes [1].
>> > >> >> >  So I don't understand why we can't do the same for the
>> make-mod-
>> > >> >> >  scripts
>> > >> >> >  who is the twin brother of all these kernel recipes.
>> > >> >> > 
>> > >> >> >  [1]
>> > >> >> > 
>> https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
>> > >> >> > >>>
>> > >> >> > >>> Ideally we wouldn't be doing this for the kernel recipes.
>> > >> >> > >>>
>> > >> >> > >>> There is also a big difference to that and the proposed
>> patch. The
>> > >> >> > >>> proposed patch was preserving a specific directory rather

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Bruce Ashfield  escreveu no dia terça, 2/05/2023
à(s) 22:12:

> Attached is v2 of the patch. I've consolidated the suggested changes.
>
> I'm soaking it a bit longer, and then will send it as part of my next
> consolidated pull request.
>

I will do some more tests with the v2 and post my comment later if anything
new comes up.

Jose


>
> Bruce
>
> On Thu, Apr 27, 2023 at 9:26 PM Bruce Ashfield via
> lists.openembedded.org
>  wrote:
> >
> > On Thu, Apr 27, 2023 at 6:32 PM Jose Quaresma 
> wrote:
> > >
> > > Hi Bruce,
> > >
> > > I have been testing your patch and have some comments.
> > > In some of my kernels I don't have the pkg-config changes and so I
> have some fails linking the scripts/sign-file
> > > because for static linking with the libcrypto we need the -ldl
> -pthread.
> > >
> > > To fix my build I need to override the CRYPTO_LIBS in my kernel
> because they use the hardcoded pkg-config
> > > where it is not possible to pass the --static argument.
> > >
> > > With following change on top of your patch I can build moist of my
> kernels:
> > >
> > > export HOSTLDFLAGS="-lz"
> > >
> > > +   HOSTPKG_CONFIG="pkg-config --static"
> > > +   # override CRYPTO_LIBS since HOSTPKG_CONFIG lands only in
> v5.19-rc1
> > > +   #
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
> > > +   CRYPTO_LIBS="$(pkg-config --static --libs libcrypto
> 2>/dev/null || echo -lcrypto)"
> > > +
> > > unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> > > for t in prepare scripts_basic scripts; do
> > > oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
> > > AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
> > > -   HOSTPKG_CONFIG="pkg-config --static" \
> > > +   HOSTPKG_CONFIG="${HOSTPKG_CONFIG}"
> CRYPTO_LIBS="${CRYPTO_LIBS}" \
> > > -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR}
> $t
> > > done
> > >
> > >
> > > I think belive that the LIBELF_LIBS needs the same fix for the cases
> where HOSTPKG_CONFIG is not available.
> > > Also I think there is a typo in the LIBELF_LIBS because you first
> populate it with pkg-config but on the export the variable is redefined
> > > with the HOST_LIBELF_LIBS. I made a litle change too on that:
> > >
> > > # for pre-5.15 kernels
> > > -   LIBELF_LIBS=$(pkg-config libelf --libs 2>/dev/null || echo
> -lelf)
> > > -   export LIBELF_LIBS="$HOST_LIBELF_LIBS -lz"
> > > +   LIBELF_LIBS=$(pkg-config --static libelf --libs 2>/dev/null ||
> echo -lelf)
> > > +   export LIBELF_LIBS="$LIBELF_LIBS -lz"
> > > export HOSTLDFLAGS="-lz"
> > >
> > > Thanks for you help.
> >
> > Those are definitely plausible tweaks to the patch, I was providing
> > the two techniques for that reason, and you've used them
> > appropriately.
> >
> > Let me roll your changes into my patch, re-test and I'll submit it to
> > the mailing list as a v2.
> >
> > Thanks for the testing, and fixup, I knew there would be things missing!
> :)
> >
> > Bruce
> >
> > >
> > > Jose
> > >
> > > Bruce Ashfield  escreveu no dia segunda,
> 24/04/2023 à(s) 20:25:
> > >>
> > >> On Mon, Apr 24, 2023 at 6:30 AM Jose Quaresma <
> quaresma.j...@gmail.com> wrote:
> > >> >
> > >> >
> > >> >
> > >> > Bruce Ashfield  escreveu no dia domingo,
> 23/04/2023 à(s) 20:55:
> > >> >>
> > >> >> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
> > >> >>  wrote:
> > >> >> >
> > >> >> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
> > >> >> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
> > >> >> > > lists.openembedded.org
> > >> >> > >  wrote:
> > >> >> > >>
> > >> >> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
> > >> >> > >>  wrote:
> > >> >> > >>>
> > >> >> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
> > >> >> >  Hi,
> > >> >> > 
> > >> >> >  Not related with the previous discussion but just for
> > >> >> >  your information.
> > >> >> >  The rm_work.bbclass has an exception for the kernel recipes
> [1].
> > >> >> >  So I don't understand why we can't do the same for the
> make-mod-
> > >> >> >  scripts
> > >> >> >  who is the twin brother of all these kernel recipes.
> > >> >> > 
> > >> >> >  [1]
> > >> >> > 
> https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
> > >> >> > >>>
> > >> >> > >>> Ideally we wouldn't be doing this for the kernel recipes.
> > >> >> > >>>
> > >> >> > >>> There is also a big difference to that and the proposed
> patch. The
> > >> >> > >>> proposed patch was preserving a specific directory rather
> than an
> > >> >> > >>> entire recipe. Removing the task stamps but leaving a small
> piece of
> > >> >> > >>> WORKDIR is quite different to preserving WORKDIR and STAMPS
> for a
> > >> >> > >>> specific recipe. The former is not tested and will break
> things. The
> > >> >> > >>> latter is better tolerated by 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

Attached is v2 of the patch. I've consolidated the suggested changes.

I'm soaking it a bit longer, and then will send it as part of my next
consolidated pull request.

Bruce

On Thu, Apr 27, 2023 at 9:26 PM Bruce Ashfield via
lists.openembedded.org
 wrote:
>
> On Thu, Apr 27, 2023 at 6:32 PM Jose Quaresma  wrote:
> >
> > Hi Bruce,
> >
> > I have been testing your patch and have some comments.
> > In some of my kernels I don't have the pkg-config changes and so I have 
> > some fails linking the scripts/sign-file
> > because for static linking with the libcrypto we need the -ldl -pthread.
> >
> > To fix my build I need to override the CRYPTO_LIBS in my kernel because 
> > they use the hardcoded pkg-config
> > where it is not possible to pass the --static argument.
> >
> > With following change on top of your patch I can build moist of my kernels:
> >
> > export HOSTLDFLAGS="-lz"
> >
> > +   HOSTPKG_CONFIG="pkg-config --static"
> > +   # override CRYPTO_LIBS since HOSTPKG_CONFIG lands only in v5.19-rc1
> > +   # 
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
> > +   CRYPTO_LIBS="$(pkg-config --static --libs libcrypto 2>/dev/null || 
> > echo -lcrypto)"
> > +
> > unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> > for t in prepare scripts_basic scripts; do
> > oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
> > AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
> > -   HOSTPKG_CONFIG="pkg-config --static" \
> > +   HOSTPKG_CONFIG="${HOSTPKG_CONFIG}" 
> > CRYPTO_LIBS="${CRYPTO_LIBS}" \
> > -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> > done
> >
> >
> > I think belive that the LIBELF_LIBS needs the same fix for the cases where 
> > HOSTPKG_CONFIG is not available.
> > Also I think there is a typo in the LIBELF_LIBS because you first populate 
> > it with pkg-config but on the export the variable is redefined
> > with the HOST_LIBELF_LIBS. I made a litle change too on that:
> >
> > # for pre-5.15 kernels
> > -   LIBELF_LIBS=$(pkg-config libelf --libs 2>/dev/null || echo -lelf)
> > -   export LIBELF_LIBS="$HOST_LIBELF_LIBS -lz"
> > +   LIBELF_LIBS=$(pkg-config --static libelf --libs 2>/dev/null || echo 
> > -lelf)
> > +   export LIBELF_LIBS="$LIBELF_LIBS -lz"
> > export HOSTLDFLAGS="-lz"
> >
> > Thanks for you help.
>
> Those are definitely plausible tweaks to the patch, I was providing
> the two techniques for that reason, and you've used them
> appropriately.
>
> Let me roll your changes into my patch, re-test and I'll submit it to
> the mailing list as a v2.
>
> Thanks for the testing, and fixup, I knew there would be things missing! :)
>
> Bruce
>
> >
> > Jose
> >
> > Bruce Ashfield  escreveu no dia segunda, 
> > 24/04/2023 à(s) 20:25:
> >>
> >> On Mon, Apr 24, 2023 at 6:30 AM Jose Quaresma  
> >> wrote:
> >> >
> >> >
> >> >
> >> > Bruce Ashfield  escreveu no dia domingo, 
> >> > 23/04/2023 à(s) 20:55:
> >> >>
> >> >> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
> >> >>  wrote:
> >> >> >
> >> >> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
> >> >> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
> >> >> > > lists.openembedded.org
> >> >> > >  wrote:
> >> >> > >>
> >> >> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
> >> >> > >>  wrote:
> >> >> > >>>
> >> >> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
> >> >> >  Hi,
> >> >> > 
> >> >> >  Not related with the previous discussion but just for
> >> >> >  your information.
> >> >> >  The rm_work.bbclass has an exception for the kernel recipes [1].
> >> >> >  So I don't understand why we can't do the same for the make-mod-
> >> >> >  scripts
> >> >> >  who is the twin brother of all these kernel recipes.
> >> >> > 
> >> >> >  [1]
> >> >> >  https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
> >> >> > >>>
> >> >> > >>> Ideally we wouldn't be doing this for the kernel recipes.
> >> >> > >>>
> >> >> > >>> There is also a big difference to that and the proposed patch. The
> >> >> > >>> proposed patch was preserving a specific directory rather than an
> >> >> > >>> entire recipe. Removing the task stamps but leaving a small piece 
> >> >> > >>> of
> >> >> > >>> WORKDIR is quite different to preserving WORKDIR and STAMPS for a
> >> >> > >>> specific recipe. The former is not tested and will break things. 
> >> >> > >>> The
> >> >> > >>> latter is better tolerated by bitbake.
> >> >> > >>
> >> >> > >> Agreed.
> >> >> > >>
> >> >> > >> Plus, I am working on this now.
> >> >> > >>
> >> >> > >> I have static linking of the scripts/tools working, but what I 
> >> >> > >> haven't
> >> >> > >> figured out is how to do that without patching the Makefiles.
> >> >> > >>
> >> >> > >
> >> >> > > It turned out to be quite the battle to get 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Thu, Apr 27, 2023 at 6:32 PM Jose Quaresma  wrote:
>
> Hi Bruce,
>
> I have been testing your patch and have some comments.
> In some of my kernels I don't have the pkg-config changes and so I have some 
> fails linking the scripts/sign-file
> because for static linking with the libcrypto we need the -ldl -pthread.
>
> To fix my build I need to override the CRYPTO_LIBS in my kernel because they 
> use the hardcoded pkg-config
> where it is not possible to pass the --static argument.
>
> With following change on top of your patch I can build moist of my kernels:
>
> export HOSTLDFLAGS="-lz"
>
> +   HOSTPKG_CONFIG="pkg-config --static"
> +   # override CRYPTO_LIBS since HOSTPKG_CONFIG lands only in v5.19-rc1
> +   # 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
> +   CRYPTO_LIBS="$(pkg-config --static --libs libcrypto 2>/dev/null || 
> echo -lcrypto)"
> +
> unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> for t in prepare scripts_basic scripts; do
> oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
> AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
> -   HOSTPKG_CONFIG="pkg-config --static" \
> +   HOSTPKG_CONFIG="${HOSTPKG_CONFIG}" 
> CRYPTO_LIBS="${CRYPTO_LIBS}" \
> -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> done
>
>
> I think belive that the LIBELF_LIBS needs the same fix for the cases where 
> HOSTPKG_CONFIG is not available.
> Also I think there is a typo in the LIBELF_LIBS because you first populate it 
> with pkg-config but on the export the variable is redefined
> with the HOST_LIBELF_LIBS. I made a litle change too on that:
>
> # for pre-5.15 kernels
> -   LIBELF_LIBS=$(pkg-config libelf --libs 2>/dev/null || echo -lelf)
> -   export LIBELF_LIBS="$HOST_LIBELF_LIBS -lz"
> +   LIBELF_LIBS=$(pkg-config --static libelf --libs 2>/dev/null || echo 
> -lelf)
> +   export LIBELF_LIBS="$LIBELF_LIBS -lz"
> export HOSTLDFLAGS="-lz"
>
> Thanks for you help.

Those are definitely plausible tweaks to the patch, I was providing
the two techniques for that reason, and you've used them
appropriately.

Let me roll your changes into my patch, re-test and I'll submit it to
the mailing list as a v2.

Thanks for the testing, and fixup, I knew there would be things missing! :)

Bruce

>
> Jose
>
> Bruce Ashfield  escreveu no dia segunda, 24/04/2023 
> à(s) 20:25:
>>
>> On Mon, Apr 24, 2023 at 6:30 AM Jose Quaresma  
>> wrote:
>> >
>> >
>> >
>> > Bruce Ashfield  escreveu no dia domingo, 
>> > 23/04/2023 à(s) 20:55:
>> >>
>> >> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
>> >>  wrote:
>> >> >
>> >> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
>> >> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
>> >> > > lists.openembedded.org
>> >> > >  wrote:
>> >> > >>
>> >> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
>> >> > >>  wrote:
>> >> > >>>
>> >> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
>> >> >  Hi,
>> >> > 
>> >> >  Not related with the previous discussion but just for
>> >> >  your information.
>> >> >  The rm_work.bbclass has an exception for the kernel recipes [1].
>> >> >  So I don't understand why we can't do the same for the make-mod-
>> >> >  scripts
>> >> >  who is the twin brother of all these kernel recipes.
>> >> > 
>> >> >  [1]
>> >> >  https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
>> >> > >>>
>> >> > >>> Ideally we wouldn't be doing this for the kernel recipes.
>> >> > >>>
>> >> > >>> There is also a big difference to that and the proposed patch. The
>> >> > >>> proposed patch was preserving a specific directory rather than an
>> >> > >>> entire recipe. Removing the task stamps but leaving a small piece of
>> >> > >>> WORKDIR is quite different to preserving WORKDIR and STAMPS for a
>> >> > >>> specific recipe. The former is not tested and will break things. The
>> >> > >>> latter is better tolerated by bitbake.
>> >> > >>
>> >> > >> Agreed.
>> >> > >>
>> >> > >> Plus, I am working on this now.
>> >> > >>
>> >> > >> I have static linking of the scripts/tools working, but what I 
>> >> > >> haven't
>> >> > >> figured out is how to do that without patching the Makefiles.
>> >> > >>
>> >> > >
>> >> > > It turned out to be quite the battle to get older kernels what was
>> >> > > required for static linking of the tools.
>> >> > >
>> >> > > Attached is my WIP patch. I'm out of the office early next week, but
>> >> > > will revisit it once I'm back.
>> >> > >
>> >> > > Bruce
>> >> > >
>> >> > >> Next up will be some rpath trickery.
>> >> > >>
>> >> > >> Bruce
>> >> > >>
>> >> > >>>
>> >> > >>> So yes, we could do the same. I'm sure there will be other recipes
>> >> > >>> people want to preserve for other reasons. Where do we draw the 
>> >> > >>> line?
>> >> > 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Hi Bruce,

I have been testing your patch and have some comments.
In some of my kernels I don't have the pkg-config changes and so I have
some fails linking the scripts/sign-file
because for static linking with the libcrypto we need the -ldl -pthread.

To fix my build I need to override the CRYPTO_LIBS in my kernel because
they use the hardcoded pkg-config
where it is not possible to pass the --static argument.

With following change on top of your patch I can build moist of my kernels:

export HOSTLDFLAGS="-lz"

+   HOSTPKG_CONFIG="pkg-config --static"
+   # override CRYPTO_LIBS since HOSTPKG_CONFIG lands only in v5.19-rc1
+   #
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
+   CRYPTO_LIBS="$(pkg-config --static --libs libcrypto 2>/dev/null ||
echo -lcrypto)"
+
unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
for t in prepare scripts_basic scripts; do
oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" \
AR="${KERNEL_AR}" OBJCOPY="${KERNEL_OBJCOPY}" \
-   HOSTPKG_CONFIG="pkg-config --static" \
+   HOSTPKG_CONFIG="${HOSTPKG_CONFIG}"
CRYPTO_LIBS="${CRYPTO_LIBS}" \
-C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
done


I think belive that the LIBELF_LIBS needs the same fix for the cases where
HOSTPKG_CONFIG is not available.
Also I think there is a typo in the LIBELF_LIBS because you first populate
it with pkg-config but on the export the variable is redefined
with the HOST_LIBELF_LIBS. I made a litle change too on that:

# for pre-5.15 kernels
-   LIBELF_LIBS=$(pkg-config libelf --libs 2>/dev/null || echo -lelf)
-   export LIBELF_LIBS="$HOST_LIBELF_LIBS -lz"
+   LIBELF_LIBS=$(pkg-config --static libelf --libs 2>/dev/null || echo
-lelf)
+   export LIBELF_LIBS="$LIBELF_LIBS -lz"
export HOSTLDFLAGS="-lz"

Thanks for you help.

Jose

Bruce Ashfield  escreveu no dia segunda,
24/04/2023 à(s) 20:25:

> On Mon, Apr 24, 2023 at 6:30 AM Jose Quaresma 
> wrote:
> >
> >
> >
> > Bruce Ashfield  escreveu no dia domingo,
> 23/04/2023 à(s) 20:55:
> >>
> >> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
> >>  wrote:
> >> >
> >> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
> >> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
> >> > > lists.openembedded.org
> >> > >  wrote:
> >> > >>
> >> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
> >> > >>  wrote:
> >> > >>>
> >> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
> >> >  Hi,
> >> > 
> >> >  Not related with the previous discussion but just for
> >> >  your information.
> >> >  The rm_work.bbclass has an exception for the kernel recipes [1].
> >> >  So I don't understand why we can't do the same for the make-mod-
> >> >  scripts
> >> >  who is the twin brother of all these kernel recipes.
> >> > 
> >> >  [1]
> >> > 
> https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
> >> > >>>
> >> > >>> Ideally we wouldn't be doing this for the kernel recipes.
> >> > >>>
> >> > >>> There is also a big difference to that and the proposed patch. The
> >> > >>> proposed patch was preserving a specific directory rather than an
> >> > >>> entire recipe. Removing the task stamps but leaving a small piece
> of
> >> > >>> WORKDIR is quite different to preserving WORKDIR and STAMPS for a
> >> > >>> specific recipe. The former is not tested and will break things.
> The
> >> > >>> latter is better tolerated by bitbake.
> >> > >>
> >> > >> Agreed.
> >> > >>
> >> > >> Plus, I am working on this now.
> >> > >>
> >> > >> I have static linking of the scripts/tools working, but what I
> haven't
> >> > >> figured out is how to do that without patching the Makefiles.
> >> > >>
> >> > >
> >> > > It turned out to be quite the battle to get older kernels what was
> >> > > required for static linking of the tools.
> >> > >
> >> > > Attached is my WIP patch. I'm out of the office early next week, but
> >> > > will revisit it once I'm back.
> >> > >
> >> > > Bruce
> >> > >
> >> > >> Next up will be some rpath trickery.
> >> > >>
> >> > >> Bruce
> >> > >>
> >> > >>>
> >> > >>> So yes, we could do the same. I'm sure there will be other recipes
> >> > >>> people want to preserve for other reasons. Where do we draw the
> line?
> >> > >>> We could preserve everything and drop rm_work, then we wouldn't
> have
> >> > >>> these problems? :)
> >> > >>>
> >> > >>> Cheers,
> >> > >>>
> >> > >>> Richard
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> - Thou shalt not follow the NULL pointer, for chaos and madness
> await
> >> > >> thee at its end
> >> > >> - "Use the force Harry" - Gandalf, Star Trek II
> >> > >>
> >> > >> 
> >> > >>
> >> > >
> >> > >
> >> >
> >> > Thank you for your work, I see you put some time and effort into it.
> >> > HOSTPKG_CONFIG is, as you mentioned, available since 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Mon, Apr 24, 2023 at 6:30 AM Jose Quaresma  wrote:
>
>
>
> Bruce Ashfield  escreveu no dia domingo, 23/04/2023 
> à(s) 20:55:
>>
>> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
>>  wrote:
>> >
>> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
>> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
>> > > lists.openembedded.org
>> > >  wrote:
>> > >>
>> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
>> > >>  wrote:
>> > >>>
>> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
>> >  Hi,
>> > 
>> >  Not related with the previous discussion but just for
>> >  your information.
>> >  The rm_work.bbclass has an exception for the kernel recipes [1].
>> >  So I don't understand why we can't do the same for the make-mod-
>> >  scripts
>> >  who is the twin brother of all these kernel recipes.
>> > 
>> >  [1]
>> >  https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
>> > >>>
>> > >>> Ideally we wouldn't be doing this for the kernel recipes.
>> > >>>
>> > >>> There is also a big difference to that and the proposed patch. The
>> > >>> proposed patch was preserving a specific directory rather than an
>> > >>> entire recipe. Removing the task stamps but leaving a small piece of
>> > >>> WORKDIR is quite different to preserving WORKDIR and STAMPS for a
>> > >>> specific recipe. The former is not tested and will break things. The
>> > >>> latter is better tolerated by bitbake.
>> > >>
>> > >> Agreed.
>> > >>
>> > >> Plus, I am working on this now.
>> > >>
>> > >> I have static linking of the scripts/tools working, but what I haven't
>> > >> figured out is how to do that without patching the Makefiles.
>> > >>
>> > >
>> > > It turned out to be quite the battle to get older kernels what was
>> > > required for static linking of the tools.
>> > >
>> > > Attached is my WIP patch. I'm out of the office early next week, but
>> > > will revisit it once I'm back.
>> > >
>> > > Bruce
>> > >
>> > >> Next up will be some rpath trickery.
>> > >>
>> > >> Bruce
>> > >>
>> > >>>
>> > >>> So yes, we could do the same. I'm sure there will be other recipes
>> > >>> people want to preserve for other reasons. Where do we draw the line?
>> > >>> We could preserve everything and drop rm_work, then we wouldn't have
>> > >>> these problems? :)
>> > >>>
>> > >>> Cheers,
>> > >>>
>> > >>> Richard
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> > >> thee at its end
>> > >> - "Use the force Harry" - Gandalf, Star Trek II
>> > >>
>> > >> 
>> > >>
>> > >
>> > >
>> >
>> > Thank you for your work, I see you put some time and effort into it.
>> > HOSTPKG_CONFIG is, as you mentioned, available since kernel version 5.19
>>
>> Yes, I realize that and documented it in the patch ... but I also
>> tested on pre-5.19 kernels and what I have in the patch works. Did it
>> not work in your testing ?
>
>
> I will test the patch on a couple of kernel versions with some of them 
> pre-5.19
> but all in 5 major versions.
> I will say something about my results later this week.

5.15-stable also has the pkg-config changes

Bruce

>
> Thanks for working on this one.
>
> Jose
>
>>
>>
>> > (see kernel patch [1]), so we need a way to call 'pkg-config --static'
>> > with pre-5.19 kernels. A way without modifying the Makefile would be to
>> > modify openssls pkg-config in recipe-sysroot-native of make-mod-script,
>> > so 'pkg-config --libs' actually shows the dependencies of 'pkg-config
>> > --static --libs', but it's a bit hacky.
>>
>> Already considered, and discarded. That's not going to fly.
>>
>> >
>> > Also fully-static executables still need the same glibc during runtime
>> > that they were built with, which makes them error-prone and is generally
>> > discouraged. As an alternative, we could build dynamic executables that
>> > use the static libcrypto library. The linker links by default against
>> > the shared library, so we could remove them from recipe-sysroot-native
>> > to force linking against the static library (again, somewhat hacky).
>>
>> Also considered and discarded.
>>
>> As do the dynamically linked ones for the c runtime. We aren't talking
>> about using these outside of a single build and they are generated on
>> the fly, so again, there's very little concern about runtimes changing
>> after linking.. There's less risk in static than in the alternatives.
>>
>> Bruce
>>
>>
>> >
>> > [1]
>> > https://github.com/torvalds/linux/commit/d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
>>
>>
>>
>> --
>> - Thou shalt not follow the NULL pointer, for chaos and madness await
>> thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>
>
>
> --
> Best regards,
>
> José Quaresma



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Bruce Ashfield  escreveu no dia domingo,
23/04/2023 à(s) 20:55:

> On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
>  wrote:
> >
> > Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
> > > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
> > > lists.openembedded.org
> > >  wrote:
> > >>
> > >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
> > >>  wrote:
> > >>>
> > >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
> >  Hi,
> > 
> >  Not related with the previous discussion but just for
> >  your information.
> >  The rm_work.bbclass has an exception for the kernel recipes [1].
> >  So I don't understand why we can't do the same for the make-mod-
> >  scripts
> >  who is the twin brother of all these kernel recipes.
> > 
> >  [1]
> > 
> https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
> > >>>
> > >>> Ideally we wouldn't be doing this for the kernel recipes.
> > >>>
> > >>> There is also a big difference to that and the proposed patch. The
> > >>> proposed patch was preserving a specific directory rather than an
> > >>> entire recipe. Removing the task stamps but leaving a small piece of
> > >>> WORKDIR is quite different to preserving WORKDIR and STAMPS for a
> > >>> specific recipe. The former is not tested and will break things. The
> > >>> latter is better tolerated by bitbake.
> > >>
> > >> Agreed.
> > >>
> > >> Plus, I am working on this now.
> > >>
> > >> I have static linking of the scripts/tools working, but what I haven't
> > >> figured out is how to do that without patching the Makefiles.
> > >>
> > >
> > > It turned out to be quite the battle to get older kernels what was
> > > required for static linking of the tools.
> > >
> > > Attached is my WIP patch. I'm out of the office early next week, but
> > > will revisit it once I'm back.
> > >
> > > Bruce
> > >
> > >> Next up will be some rpath trickery.
> > >>
> > >> Bruce
> > >>
> > >>>
> > >>> So yes, we could do the same. I'm sure there will be other recipes
> > >>> people want to preserve for other reasons. Where do we draw the line?
> > >>> We could preserve everything and drop rm_work, then we wouldn't have
> > >>> these problems? :)
> > >>>
> > >>> Cheers,
> > >>>
> > >>> Richard
> > >>
> > >>
> > >>
> > >> --
> > >> - Thou shalt not follow the NULL pointer, for chaos and madness await
> > >> thee at its end
> > >> - "Use the force Harry" - Gandalf, Star Trek II
> > >>
> > >> 
> > >>
> > >
> > >
> >
> > Thank you for your work, I see you put some time and effort into it.
> > HOSTPKG_CONFIG is, as you mentioned, available since kernel version 5.19
>
> Yes, I realize that and documented it in the patch ... but I also
> tested on pre-5.19 kernels and what I have in the patch works. Did it
> not work in your testing ?
>

I will test the patch on a couple of kernel versions with some of them
pre-5.19
but all in 5 major versions.
I will say something about my results later this week.

Thanks for working on this one.

Jose


>
> > (see kernel patch [1]), so we need a way to call 'pkg-config --static'
> > with pre-5.19 kernels. A way without modifying the Makefile would be to
> > modify openssls pkg-config in recipe-sysroot-native of make-mod-script,
> > so 'pkg-config --libs' actually shows the dependencies of 'pkg-config
> > --static --libs', but it's a bit hacky.
>
> Already considered, and discarded. That's not going to fly.
>
> >
> > Also fully-static executables still need the same glibc during runtime
> > that they were built with, which makes them error-prone and is generally
> > discouraged. As an alternative, we could build dynamic executables that
> > use the static libcrypto library. The linker links by default against
> > the shared library, so we could remove them from recipe-sysroot-native
> > to force linking against the static library (again, somewhat hacky).
>
> Also considered and discarded.
>
> As do the dynamically linked ones for the c runtime. We aren't talking
> about using these outside of a single build and they are generated on
> the fly, so again, there's very little concern about runtimes changing
> after linking.. There's less risk in static than in the alternatives.
>
> Bruce
>
>
> >
> > [1]
> >
> https://github.com/torvalds/linux/commit/d5ea4fece4508bf8e72b659cd22fa4840d8d61e5
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>


-- 
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180349): 
https://lists.openembedded.org/g/openembedded-core/message/180349
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Sat, Apr 22, 2023 at 9:06 AM Christoph Lauer
 wrote:
>
> Am 21.04.23 um 22:28 schrieb Bruce Ashfield:
> > On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
> > lists.openembedded.org
> >  wrote:
> >>
> >> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
> >>  wrote:
> >>>
> >>> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
>  Hi,
> 
>  Not related with the previous discussion but just for
>  your information.
>  The rm_work.bbclass has an exception for the kernel recipes [1].
>  So I don't understand why we can't do the same for the make-mod-
>  scripts
>  who is the twin brother of all these kernel recipes.
> 
>  [1]
>  https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
> >>>
> >>> Ideally we wouldn't be doing this for the kernel recipes.
> >>>
> >>> There is also a big difference to that and the proposed patch. The
> >>> proposed patch was preserving a specific directory rather than an
> >>> entire recipe. Removing the task stamps but leaving a small piece of
> >>> WORKDIR is quite different to preserving WORKDIR and STAMPS for a
> >>> specific recipe. The former is not tested and will break things. The
> >>> latter is better tolerated by bitbake.
> >>
> >> Agreed.
> >>
> >> Plus, I am working on this now.
> >>
> >> I have static linking of the scripts/tools working, but what I haven't
> >> figured out is how to do that without patching the Makefiles.
> >>
> >
> > It turned out to be quite the battle to get older kernels what was
> > required for static linking of the tools.
> >
> > Attached is my WIP patch. I'm out of the office early next week, but
> > will revisit it once I'm back.
> >
> > Bruce
> >
> >> Next up will be some rpath trickery.
> >>
> >> Bruce
> >>
> >>>
> >>> So yes, we could do the same. I'm sure there will be other recipes
> >>> people want to preserve for other reasons. Where do we draw the line?
> >>> We could preserve everything and drop rm_work, then we wouldn't have
> >>> these problems? :)
> >>>
> >>> Cheers,
> >>>
> >>> Richard
> >>
> >>
> >>
> >> --
> >> - Thou shalt not follow the NULL pointer, for chaos and madness await
> >> thee at its end
> >> - "Use the force Harry" - Gandalf, Star Trek II
> >>
> >> 
> >>
> >
> >
>
> Thank you for your work, I see you put some time and effort into it.
> HOSTPKG_CONFIG is, as you mentioned, available since kernel version 5.19

Yes, I realize that and documented it in the patch ... but I also
tested on pre-5.19 kernels and what I have in the patch works. Did it
not work in your testing ?

> (see kernel patch [1]), so we need a way to call 'pkg-config --static'
> with pre-5.19 kernels. A way without modifying the Makefile would be to
> modify openssls pkg-config in recipe-sysroot-native of make-mod-script,
> so 'pkg-config --libs' actually shows the dependencies of 'pkg-config
> --static --libs', but it's a bit hacky.

Already considered, and discarded. That's not going to fly.

>
> Also fully-static executables still need the same glibc during runtime
> that they were built with, which makes them error-prone and is generally
> discouraged. As an alternative, we could build dynamic executables that
> use the static libcrypto library. The linker links by default against
> the shared library, so we could remove them from recipe-sysroot-native
> to force linking against the static library (again, somewhat hacky).

Also considered and discarded.

As do the dynamically linked ones for the c runtime. We aren't talking
about using these outside of a single build and they are generated on
the fly, so again, there's very little concern about runtimes changing
after linking.. There's less risk in static than in the alternatives.

Bruce


>
> [1]
> https://github.com/torvalds/linux/commit/d5ea4fece4508bf8e72b659cd22fa4840d8d61e5



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180336): 
https://lists.openembedded.org/g/openembedded-core/message/180336
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Christoph Lauer]

Am 21.04.23 um 22:28 schrieb Bruce Ashfield:

On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
lists.openembedded.org
 wrote:


On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
 wrote:


On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:

Hi,

Not related with the previous discussion but just for
your information.
The rm_work.bbclass has an exception for the kernel recipes [1].
So I don't understand why we can't do the same for the make-mod-
scripts
who is the twin brother of all these kernel recipes.

[1]
https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168


Ideally we wouldn't be doing this for the kernel recipes.

There is also a big difference to that and the proposed patch. The
proposed patch was preserving a specific directory rather than an
entire recipe. Removing the task stamps but leaving a small piece of
WORKDIR is quite different to preserving WORKDIR and STAMPS for a
specific recipe. The former is not tested and will break things. The
latter is better tolerated by bitbake.


Agreed.

Plus, I am working on this now.

I have static linking of the scripts/tools working, but what I haven't
figured out is how to do that without patching the Makefiles.



It turned out to be quite the battle to get older kernels what was
required for static linking of the tools.

Attached is my WIP patch. I'm out of the office early next week, but
will revisit it once I'm back.

Bruce


Next up will be some rpath trickery.

Bruce



So yes, we could do the same. I'm sure there will be other recipes
people want to preserve for other reasons. Where do we draw the line?
We could preserve everything and drop rm_work, then we wouldn't have
these problems? :)

Cheers,

Richard




--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II








Thank you for your work, I see you put some time and effort into it.
HOSTPKG_CONFIG is, as you mentioned, available since kernel version 5.19
(see kernel patch [1]), so we need a way to call 'pkg-config --static'
with pre-5.19 kernels. A way without modifying the Makefile would be to
modify openssls pkg-config in recipe-sysroot-native of make-mod-script,
so 'pkg-config --libs' actually shows the dependencies of 'pkg-config
--static --libs', but it's a bit hacky.

Also fully-static executables still need the same glibc during runtime
that they were built with, which makes them error-prone and is generally
discouraged. As an alternative, we could build dynamic executables that
use the static libcrypto library. The linker links by default against
the shared library, so we could remove them from recipe-sysroot-native
to force linking against the static library (again, somewhat hacky).

[1]
https://github.com/torvalds/linux/commit/d5ea4fece4508bf8e72b659cd22fa4840d8d61e5

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180301): 
https://lists.openembedded.org/g/openembedded-core/message/180301
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Wed, Apr 19, 2023 at 11:03 PM Bruce Ashfield via
lists.openembedded.org
 wrote:
>
> On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
>  wrote:
> >
> > On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
> > > Hi,
> > >
> > > Not related with the previous discussion but just for
> > > your information.
> > > The rm_work.bbclass has an exception for the kernel recipes [1].
> > > So I don't understand why we can't do the same for the make-mod-
> > > scripts
> > > who is the twin brother of all these kernel recipes.
> > >
> > > [1]
> > > https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
> >
> > Ideally we wouldn't be doing this for the kernel recipes.
> >
> > There is also a big difference to that and the proposed patch. The
> > proposed patch was preserving a specific directory rather than an
> > entire recipe. Removing the task stamps but leaving a small piece of
> > WORKDIR is quite different to preserving WORKDIR and STAMPS for a
> > specific recipe. The former is not tested and will break things. The
> > latter is better tolerated by bitbake.
>
> Agreed.
>
> Plus, I am working on this now.
>
> I have static linking of the scripts/tools working, but what I haven't
> figured out is how to do that without patching the Makefiles.
>

It turned out to be quite the battle to get older kernels what was
required for static linking of the tools.

Attached is my WIP patch. I'm out of the office early next week, but
will revisit it once I'm back.

Bruce

> Next up will be some rpath trickery.
>
> Bruce
>
> >
> > So yes, we could do the same. I'm sure there will be other recipes
> > people want to preserve for other reasons. Where do we draw the line?
> > We could preserve everything and drop rm_work, then we wouldn't have
> > these problems? :)
> >
> > Cheers,
> >
> > Richard
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II


0001-make-mod-scripts-force-static-linking-and-make-depen.patch
Description: Binary data

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180296): 
https://lists.openembedded.org/g/openembedded-core/message/180296
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Wed, Apr 19, 2023 at 6:54 PM Richard Purdie
 wrote:
>
> On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
> > Hi,
> >
> > Not related with the previous discussion but just for
> > your information.
> > The rm_work.bbclass has an exception for the kernel recipes [1].
> > So I don't understand why we can't do the same for the make-mod-
> > scripts
> > who is the twin brother of all these kernel recipes.
> >
> > [1]
> > https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168
>
> Ideally we wouldn't be doing this for the kernel recipes.
>
> There is also a big difference to that and the proposed patch. The
> proposed patch was preserving a specific directory rather than an
> entire recipe. Removing the task stamps but leaving a small piece of
> WORKDIR is quite different to preserving WORKDIR and STAMPS for a
> specific recipe. The former is not tested and will break things. The
> latter is better tolerated by bitbake.

Agreed.

Plus, I am working on this now.

I have static linking of the scripts/tools working, but what I haven't
figured out is how to do that without patching the Makefiles.

Next up will be some rpath trickery.

Bruce

>
> So yes, we could do the same. I'm sure there will be other recipes
> people want to preserve for other reasons. Where do we draw the line?
> We could preserve everything and drop rm_work, then we wouldn't have
> these problems? :)
>
> Cheers,
>
> Richard



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180233): 
https://lists.openembedded.org/g/openembedded-core/message/180233
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Richard Purdie]

On Wed, 2023-04-19 at 23:34 +0100, Jose Quaresma wrote:
> Hi,
> 
> Not related with the previous discussion but just for
> your information.
> The rm_work.bbclass has an exception for the kernel recipes [1].
> So I don't understand why we can't do the same for the make-mod-
> scripts
> who is the twin brother of all these kernel recipes.
> 
> [1]
> https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168

Ideally we wouldn't be doing this for the kernel recipes.

There is also a big difference to that and the proposed patch. The
proposed patch was preserving a specific directory rather than an
entire recipe. Removing the task stamps but leaving a small piece of
WORKDIR is quite different to preserving WORKDIR and STAMPS for a
specific recipe. The former is not tested and will break things. The
latter is better tolerated by bitbake.

So yes, we could do the same. I'm sure there will be other recipes
people want to preserve for other reasons. Where do we draw the line?
We could preserve everything and drop rm_work, then we wouldn't have
these problems? :)

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180231): 
https://lists.openembedded.org/g/openembedded-core/message/180231
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Hi,

Not related with the previous discussion but just for your information.
The rm_work.bbclass has an exception for the kernel recipes [1].
So I don't understand why we can't do the same for the make-mod-scripts
who is the twin brother of all these kernel recipes.

[1]
https://git.openembedded.org/openembedded-core/tree/meta/classes/rm_work.bbclass#n168

Jose


Bruce Ashfield  escreveu no dia quarta,
19/04/2023 à(s) 14:59:

> On Wed, Apr 19, 2023 at 9:52 AM Jose Quaresma 
> wrote:
> >
> >
> >
> > Bruce Ashfield  escreveu no dia terça,
> 18/04/2023 à(s) 22:12:
> >>
> >> On Tue, Apr 18, 2023 at 5:04 PM Richard Purdie
> >>  wrote:
> >> >
> >> > On Tue, 2023-04-18 at 16:25 -0400, Bruce Ashfield wrote:
> >> > > On Mon, Apr 17, 2023 at 6:31 PM Jose Quaresma <
> quaresma.j...@gmail.com> wrote:
> >> > > >
> >> > > >
> >> > > >
> >> > > > Richard Purdie  escreveu no
> dia segunda, 17/04/2023 à(s) 20:51:
> >> > > > >
> >> > > > > On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
> >> > > > > > From: Christoph Lauer 
> >> > > > > >
> >> > > > > > With rm_work active, external module signing throws an error:
> >> > > > > > scripts/sign-file: error while loading shared libraries:
> libcrypto.so.3: cannot open shared object file: No such file or directory
> >> > > > > > Preserve libraries that sign-file script needs during runtime.
> >> > > > > >
> >> > > > > > Signed-off-by: Christoph Lauer 
> >> > > > > > ---
> >> > > > > >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> | 3 +++
> >> > > > > >  1 file changed, 3 insertions(+)
> >> > > > > >
> >> > > > > > diff --git a/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb
> >> > > > > > index 28e0807d1d..0e24efc597 100644
> >> > > > > > --- a/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb
> >> > > > > > +++ b/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb
> >> > > > > > @@ -32,3 +32,6 @@ do_configure() {
> >> > > > > >   -C ${STAGING_KERNEL_DIR}
> O=${STAGING_KERNEL_BUILDDIR} $t
> >> > > > > >   done
> >> > > > > >  }
> >> > > > > > +
> >> > > > > > +# keep native libraries required for module signing
> >> > > > > > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
> >> > > > >
> >> > > > > I'm really reluctant to take this change as it isn't the way
> >> > > > > dependencies are meant to work.
> >> > > > >
> >> > > > > It sounds like something in a shared workdir is depending on
> something
> >> > > > > in a recipe workdir and we simply don't support that.
> Everything needed
> >> > > > > should be in the shared workdir. At best this is a bandaid and
> there
> >> > > > > will be other ways to make this fail such as cleaning
> make-mod-scripts.
> >> > > >
> >> > > >
> >> > > > The problem is because for signing the kernel modules the
> sign-file.c [1] is linked dynamically with openssl-native.
> >> > > > This works when building the in tree kernel modules but will fail
> when we try to sing any out of tree kernel module.
> >> > > > To sign the out of tree kernel we will use the binaries from the
> shared workdir but the native libcrypto.so.3 is removed by
> >> > > > the rm_work bbclass. We need to link the sign-file statically
> otherwise it will not work with the rm_work bbclass.
> >> > > >
> >> > > > [1]
> https://github.com/torvalds/linux/blob/master/scripts/sign-file.c
> >> > > >
> >> > > > Another solution for this problem can be changing the
> make-mod-scripts to be a native tool and in this way
> >> > > > they will be installed and the dependencies will be handled
> correctly.
> >> > >
> >> > > There would very likely be different issues if the scripts were
> >> > > generated and then packaged as a native tool / package. Since they
> are
> >> > > so tightly coupled to the kernel. We'd just trade one set of issues
> >> > > for another (out of sync artifacts, etc).
> >
> >
> > Maybe some new issues will come with this change but this will be more
> aligned with
> > majority of the other tools. This is something I will keep in my TODO
> list.
> >
> >>
> >> > >
> >> > > I'm going to hack on this a bit.
> >> > >
> >> > > That being said, I've never done any module signing .. since I don't
> >> > > need it in my development workflow.
> >> > >
> >> > > Is there a canonical guide to getting it setup so I can test my
> static
> >> > > link and relocated artifacts fixes ? is it with meta-integrity and
> the
> >> > > kernel-modsign bbclass ?
> >
> >
> > Yes, I am using the kernel-modsign bbclass of meta-security.
> > The step to needed is add modsign in DISTRO_FUTURES and
> > provide the required keys setting the MODSIGN_* variable in the bbclass
> >
> >> >
> >> > I did think about this a bit more. It does likely depend on the
> version
> >> > of libcrypto from the host system as to whether it reproduces or not.
> >> > The possible solution ideas I came up with are:
> >> >
> >> > a) statically link sign-file so we don't 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Wed, Apr 19, 2023 at 9:52 AM Jose Quaresma  wrote:
>
>
>
> Bruce Ashfield  escreveu no dia terça, 18/04/2023 
> à(s) 22:12:
>>
>> On Tue, Apr 18, 2023 at 5:04 PM Richard Purdie
>>  wrote:
>> >
>> > On Tue, 2023-04-18 at 16:25 -0400, Bruce Ashfield wrote:
>> > > On Mon, Apr 17, 2023 at 6:31 PM Jose Quaresma  
>> > > wrote:
>> > > >
>> > > >
>> > > >
>> > > > Richard Purdie  escreveu no dia 
>> > > > segunda, 17/04/2023 à(s) 20:51:
>> > > > >
>> > > > > On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
>> > > > > > From: Christoph Lauer 
>> > > > > >
>> > > > > > With rm_work active, external module signing throws an error:
>> > > > > > scripts/sign-file: error while loading shared libraries: 
>> > > > > > libcrypto.so.3: cannot open shared object file: No such file or 
>> > > > > > directory
>> > > > > > Preserve libraries that sign-file script needs during runtime.
>> > > > > >
>> > > > > > Signed-off-by: Christoph Lauer 
>> > > > > > ---
>> > > > > >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 
>> > > > > > +++
>> > > > > >  1 file changed, 3 insertions(+)
>> > > > > >
>> > > > > > diff --git 
>> > > > > > a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
>> > > > > > b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
>> > > > > > index 28e0807d1d..0e24efc597 100644
>> > > > > > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
>> > > > > > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
>> > > > > > @@ -32,3 +32,6 @@ do_configure() {
>> > > > > >   -C ${STAGING_KERNEL_DIR} 
>> > > > > > O=${STAGING_KERNEL_BUILDDIR} $t
>> > > > > >   done
>> > > > > >  }
>> > > > > > +
>> > > > > > +# keep native libraries required for module signing
>> > > > > > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
>> > > > >
>> > > > > I'm really reluctant to take this change as it isn't the way
>> > > > > dependencies are meant to work.
>> > > > >
>> > > > > It sounds like something in a shared workdir is depending on 
>> > > > > something
>> > > > > in a recipe workdir and we simply don't support that. Everything 
>> > > > > needed
>> > > > > should be in the shared workdir. At best this is a bandaid and there
>> > > > > will be other ways to make this fail such as cleaning 
>> > > > > make-mod-scripts.
>> > > >
>> > > >
>> > > > The problem is because for signing the kernel modules the sign-file.c 
>> > > > [1] is linked dynamically with openssl-native.
>> > > > This works when building the in tree kernel modules but will fail when 
>> > > > we try to sing any out of tree kernel module.
>> > > > To sign the out of tree kernel we will use the binaries from the 
>> > > > shared workdir but the native libcrypto.so.3 is removed by
>> > > > the rm_work bbclass. We need to link the sign-file statically 
>> > > > otherwise it will not work with the rm_work bbclass.
>> > > >
>> > > > [1] https://github.com/torvalds/linux/blob/master/scripts/sign-file.c
>> > > >
>> > > > Another solution for this problem can be changing the make-mod-scripts 
>> > > > to be a native tool and in this way
>> > > > they will be installed and the dependencies will be handled correctly.
>> > >
>> > > There would very likely be different issues if the scripts were
>> > > generated and then packaged as a native tool / package. Since they are
>> > > so tightly coupled to the kernel. We'd just trade one set of issues
>> > > for another (out of sync artifacts, etc).
>
>
> Maybe some new issues will come with this change but this will be more 
> aligned with
> majority of the other tools. This is something I will keep in my TODO list.
>
>>
>> > >
>> > > I'm going to hack on this a bit.
>> > >
>> > > That being said, I've never done any module signing .. since I don't
>> > > need it in my development workflow.
>> > >
>> > > Is there a canonical guide to getting it setup so I can test my static
>> > > link and relocated artifacts fixes ? is it with meta-integrity and the
>> > > kernel-modsign bbclass ?
>
>
> Yes, I am using the kernel-modsign bbclass of meta-security.
> The step to needed is add modsign in DISTRO_FUTURES and
> provide the required keys setting the MODSIGN_* variable in the bbclass
>
>> >
>> > I did think about this a bit more. It does likely depend on the version
>> > of libcrypto from the host system as to whether it reproduces or not.
>> > The possible solution ideas I came up with are:
>> >
>> > a) statically link sign-file so we don't need libcrypto
>> > b) copy the libcrypto.so into a known location in the shared workdir
>> > (probably some new path) and then adding an RPATH/RUNPATH using chrpath
>> > to the binary.
>>
>> Agreed. I have sign-file statically building here, and it works, but
>> objtool is blowing up under static linking.
>
>
> Building the sign-file statically looks like a good solution but I have no 
> idea of the side effects.

There's no side effects to the tool itself (outside of the normal "the
binary is a 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Bruce Ashfield  escreveu no dia terça, 18/04/2023
à(s) 22:12:

> On Tue, Apr 18, 2023 at 5:04 PM Richard Purdie
>  wrote:
> >
> > On Tue, 2023-04-18 at 16:25 -0400, Bruce Ashfield wrote:
> > > On Mon, Apr 17, 2023 at 6:31 PM Jose Quaresma 
> wrote:
> > > >
> > > >
> > > >
> > > > Richard Purdie  escreveu no dia
> segunda, 17/04/2023 à(s) 20:51:
> > > > >
> > > > > On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
> > > > > > From: Christoph Lauer 
> > > > > >
> > > > > > With rm_work active, external module signing throws an error:
> > > > > > scripts/sign-file: error while loading shared libraries:
> libcrypto.so.3: cannot open shared object file: No such file or directory
> > > > > > Preserve libraries that sign-file script needs during runtime.
> > > > > >
> > > > > > Signed-off-by: Christoph Lauer 
> > > > > > ---
> > > > > >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb |
> 3 +++
> > > > > >  1 file changed, 3 insertions(+)
> > > > > >
> > > > > > diff --git a/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb
> > > > > > index 28e0807d1d..0e24efc597 100644
> > > > > > --- a/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb
> > > > > > +++ b/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb
> > > > > > @@ -32,3 +32,6 @@ do_configure() {
> > > > > >   -C ${STAGING_KERNEL_DIR}
> O=${STAGING_KERNEL_BUILDDIR} $t
> > > > > >   done
> > > > > >  }
> > > > > > +
> > > > > > +# keep native libraries required for module signing
> > > > > > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
> > > > >
> > > > > I'm really reluctant to take this change as it isn't the way
> > > > > dependencies are meant to work.
> > > > >
> > > > > It sounds like something in a shared workdir is depending on
> something
> > > > > in a recipe workdir and we simply don't support that. Everything
> needed
> > > > > should be in the shared workdir. At best this is a bandaid and
> there
> > > > > will be other ways to make this fail such as cleaning
> make-mod-scripts.
> > > >
> > > >
> > > > The problem is because for signing the kernel modules the
> sign-file.c [1] is linked dynamically with openssl-native.
> > > > This works when building the in tree kernel modules but will fail
> when we try to sing any out of tree kernel module.
> > > > To sign the out of tree kernel we will use the binaries from the
> shared workdir but the native libcrypto.so.3 is removed by
> > > > the rm_work bbclass. We need to link the sign-file statically
> otherwise it will not work with the rm_work bbclass.
> > > >
> > > > [1]
> https://github.com/torvalds/linux/blob/master/scripts/sign-file.c
> > > >
> > > > Another solution for this problem can be changing the
> make-mod-scripts to be a native tool and in this way
> > > > they will be installed and the dependencies will be handled
> correctly.
> > >
> > > There would very likely be different issues if the scripts were
> > > generated and then packaged as a native tool / package. Since they are
> > > so tightly coupled to the kernel. We'd just trade one set of issues
> > > for another (out of sync artifacts, etc).
>

Maybe some new issues will come with this change but this will be more
aligned with
majority of the other tools. This is something I will keep in my TODO list.


> > >
> > > I'm going to hack on this a bit.
> > >
> > > That being said, I've never done any module signing .. since I don't
> > > need it in my development workflow.
> > >
> > > Is there a canonical guide to getting it setup so I can test my static
> > > link and relocated artifacts fixes ? is it with meta-integrity and the
> > > kernel-modsign bbclass ?
>

Yes, I am using the kernel-modsign bbclass of meta-security.
The step to needed is add modsign in DISTRO_FUTURES and
provide the required keys setting the MODSIGN_* variable in the bbclass

>
> > I did think about this a bit more. It does likely depend on the version
> > of libcrypto from the host system as to whether it reproduces or not.
> > The possible solution ideas I came up with are:
> >
> > a) statically link sign-file so we don't need libcrypto
> > b) copy the libcrypto.so into a known location in the shared workdir
> > (probably some new path) and then adding an RPATH/RUNPATH using chrpath
> > to the binary.
>
> Agreed. I have sign-file statically building here, and it works, but
> objtool is blowing up under static linking.
>

Building the sign-file statically looks like a good solution but I have no
idea of the side effects.


>
> If I can't break the tools into separate bits, or fix that static link
> .. My other idea is the same as yours, if we copy out the .so and make
> sure it is in a recognized location in the artifacts dir, we are good
> to go.
>

But I believe that for copying it will require copying the full native
sysroot dependency chain and
not only libcrypto.so.

Jose


>
> Bruce
>
> >
> > Cheers,
> >
> > 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Tue, Apr 18, 2023 at 5:04 PM Richard Purdie
 wrote:
>
> On Tue, 2023-04-18 at 16:25 -0400, Bruce Ashfield wrote:
> > On Mon, Apr 17, 2023 at 6:31 PM Jose Quaresma  
> > wrote:
> > >
> > >
> > >
> > > Richard Purdie  escreveu no dia 
> > > segunda, 17/04/2023 à(s) 20:51:
> > > >
> > > > On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
> > > > > From: Christoph Lauer 
> > > > >
> > > > > With rm_work active, external module signing throws an error:
> > > > > scripts/sign-file: error while loading shared libraries: 
> > > > > libcrypto.so.3: cannot open shared object file: No such file or 
> > > > > directory
> > > > > Preserve libraries that sign-file script needs during runtime.
> > > > >
> > > > > Signed-off-by: Christoph Lauer 
> > > > > ---
> > > > >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
> > > > >  1 file changed, 3 insertions(+)
> > > > >
> > > > > diff --git 
> > > > > a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
> > > > > b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > > index 28e0807d1d..0e24efc597 100644
> > > > > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > > @@ -32,3 +32,6 @@ do_configure() {
> > > > >   -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> > > > >   done
> > > > >  }
> > > > > +
> > > > > +# keep native libraries required for module signing
> > > > > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
> > > >
> > > > I'm really reluctant to take this change as it isn't the way
> > > > dependencies are meant to work.
> > > >
> > > > It sounds like something in a shared workdir is depending on something
> > > > in a recipe workdir and we simply don't support that. Everything needed
> > > > should be in the shared workdir. At best this is a bandaid and there
> > > > will be other ways to make this fail such as cleaning make-mod-scripts.
> > >
> > >
> > > The problem is because for signing the kernel modules the sign-file.c [1] 
> > > is linked dynamically with openssl-native.
> > > This works when building the in tree kernel modules but will fail when we 
> > > try to sing any out of tree kernel module.
> > > To sign the out of tree kernel we will use the binaries from the shared 
> > > workdir but the native libcrypto.so.3 is removed by
> > > the rm_work bbclass. We need to link the sign-file statically otherwise 
> > > it will not work with the rm_work bbclass.
> > >
> > > [1] https://github.com/torvalds/linux/blob/master/scripts/sign-file.c
> > >
> > > Another solution for this problem can be changing the make-mod-scripts to 
> > > be a native tool and in this way
> > > they will be installed and the dependencies will be handled correctly.
> >
> > There would very likely be different issues if the scripts were
> > generated and then packaged as a native tool / package. Since they are
> > so tightly coupled to the kernel. We'd just trade one set of issues
> > for another (out of sync artifacts, etc).
> >
> > I'm going to hack on this a bit.
> >
> > That being said, I've never done any module signing .. since I don't
> > need it in my development workflow.
> >
> > Is there a canonical guide to getting it setup so I can test my static
> > link and relocated artifacts fixes ? is it with meta-integrity and the
> > kernel-modsign bbclass ?
>
> I did think about this a bit more. It does likely depend on the version
> of libcrypto from the host system as to whether it reproduces or not.
> The possible solution ideas I came up with are:
>
> a) statically link sign-file so we don't need libcrypto
> b) copy the libcrypto.so into a known location in the shared workdir
> (probably some new path) and then adding an RPATH/RUNPATH using chrpath
> to the binary.

Agreed. I have sign-file statically building here, and it works, but
objtool is blowing up under static linking.

If I can't break the tools into separate bits, or fix that static link
.. My other idea is the same as yours, if we copy out the .so and make
sure it is in a recognized location in the artifacts dir, we are good
to go.

Bruce

>
> Cheers,
>
> Richard
>
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180203): 
https://lists.openembedded.org/g/openembedded-core/message/180203
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Richard Purdie]

On Tue, 2023-04-18 at 16:25 -0400, Bruce Ashfield wrote:
> On Mon, Apr 17, 2023 at 6:31 PM Jose Quaresma  wrote:
> > 
> > 
> > 
> > Richard Purdie  escreveu no dia 
> > segunda, 17/04/2023 à(s) 20:51:
> > > 
> > > On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
> > > > From: Christoph Lauer 
> > > > 
> > > > With rm_work active, external module signing throws an error:
> > > > scripts/sign-file: error while loading shared libraries: 
> > > > libcrypto.so.3: cannot open shared object file: No such file or 
> > > > directory
> > > > Preserve libraries that sign-file script needs during runtime.
> > > > 
> > > > Signed-off-by: Christoph Lauer 
> > > > ---
> > > >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
> > > >  1 file changed, 3 insertions(+)
> > > > 
> > > > diff --git 
> > > > a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
> > > > b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > index 28e0807d1d..0e24efc597 100644
> > > > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > @@ -32,3 +32,6 @@ do_configure() {
> > > >   -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> > > >   done
> > > >  }
> > > > +
> > > > +# keep native libraries required for module signing
> > > > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
> > > 
> > > I'm really reluctant to take this change as it isn't the way
> > > dependencies are meant to work.
> > > 
> > > It sounds like something in a shared workdir is depending on something
> > > in a recipe workdir and we simply don't support that. Everything needed
> > > should be in the shared workdir. At best this is a bandaid and there
> > > will be other ways to make this fail such as cleaning make-mod-scripts.
> > 
> > 
> > The problem is because for signing the kernel modules the sign-file.c [1] 
> > is linked dynamically with openssl-native.
> > This works when building the in tree kernel modules but will fail when we 
> > try to sing any out of tree kernel module.
> > To sign the out of tree kernel we will use the binaries from the shared 
> > workdir but the native libcrypto.so.3 is removed by
> > the rm_work bbclass. We need to link the sign-file statically otherwise it 
> > will not work with the rm_work bbclass.
> > 
> > [1] https://github.com/torvalds/linux/blob/master/scripts/sign-file.c
> > 
> > Another solution for this problem can be changing the make-mod-scripts to 
> > be a native tool and in this way
> > they will be installed and the dependencies will be handled correctly.
> 
> There would very likely be different issues if the scripts were
> generated and then packaged as a native tool / package. Since they are
> so tightly coupled to the kernel. We'd just trade one set of issues
> for another (out of sync artifacts, etc).
> 
> I'm going to hack on this a bit.
> 
> That being said, I've never done any module signing .. since I don't
> need it in my development workflow.
> 
> Is there a canonical guide to getting it setup so I can test my static
> link and relocated artifacts fixes ? is it with meta-integrity and the
> kernel-modsign bbclass ?

I did think about this a bit more. It does likely depend on the version
of libcrypto from the host system as to whether it reproduces or not.
The possible solution ideas I came up with are:

a) statically link sign-file so we don't need libcrypto
b) copy the libcrypto.so into a known location in the shared workdir
(probably some new path) and then adding an RPATH/RUNPATH using chrpath
to the binary.

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180202): 
https://lists.openembedded.org/g/openembedded-core/message/180202
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Tue, Apr 18, 2023 at 4:25 PM Bruce Ashfield via
lists.openembedded.org
 wrote:
>
> On Mon, Apr 17, 2023 at 6:31 PM Jose Quaresma  wrote:
> >
> >
> >
> > Richard Purdie  escreveu no dia 
> > segunda, 17/04/2023 à(s) 20:51:
> >>
> >> On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
> >> > From: Christoph Lauer 
> >> >
> >> > With rm_work active, external module signing throws an error:
> >> > scripts/sign-file: error while loading shared libraries: libcrypto.so.3: 
> >> > cannot open shared object file: No such file or directory
> >> > Preserve libraries that sign-file script needs during runtime.
> >> >
> >> > Signed-off-by: Christoph Lauer 
> >> > ---
> >> >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
> >> >  1 file changed, 3 insertions(+)
> >> >
> >> > diff --git 
> >> > a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
> >> > b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> >> > index 28e0807d1d..0e24efc597 100644
> >> > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> >> > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> >> > @@ -32,3 +32,6 @@ do_configure() {
> >> >   -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> >> >   done
> >> >  }
> >> > +
> >> > +# keep native libraries required for module signing
> >> > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
> >>
> >> I'm really reluctant to take this change as it isn't the way
> >> dependencies are meant to work.
> >>
> >> It sounds like something in a shared workdir is depending on something
> >> in a recipe workdir and we simply don't support that. Everything needed
> >> should be in the shared workdir. At best this is a bandaid and there
> >> will be other ways to make this fail such as cleaning make-mod-scripts.
> >
> >
> > The problem is because for signing the kernel modules the sign-file.c [1] 
> > is linked dynamically with openssl-native.
> > This works when building the in tree kernel modules but will fail when we 
> > try to sing any out of tree kernel module.
> > To sign the out of tree kernel we will use the binaries from the shared 
> > workdir but the native libcrypto.so.3 is removed by
> > the rm_work bbclass. We need to link the sign-file statically otherwise it 
> > will not work with the rm_work bbclass.
> >
> > [1] https://github.com/torvalds/linux/blob/master/scripts/sign-file.c
> >
> > Another solution for this problem can be changing the make-mod-scripts to 
> > be a native tool and in this way
> > they will be installed and the dependencies will be handled correctly.
>
> There would very likely be different issues if the scripts were
> generated and then packaged as a native tool / package. Since they are
> so tightly coupled to the kernel. We'd just trade one set of issues
> for another (out of sync artifacts, etc).
>
> I'm going to hack on this a bit.
>
> That being said, I've never done any module signing .. since I don't
> need it in my development workflow.
>
> Is there a canonical guide to getting it setup so I can test my static
> link and relocated artifacts fixes ? is it with meta-integrity and the
> kernel-modsign bbclass ?

or are you maining just using the force-signing fragments (or
equivalent) kernel configuration ?

Bruce

>
> Bruce
>
>
> Bruce
>
> >
> >>
> >> I'm even less keen to take it when I think it's going to be backported
> >> "everywhere" as if is the correct solution too.
> >>
> >> I don't know what the right fix is unfortunately. I'm sure people would
> >> like me to think about it and come up with one but there are simply too
> >> many different things people would like me to do that with and even for
> >> me, it does take a while to work these things out. I'm just out of
> >> bandwidth, sorry :(
> >
> >
> > It is true that it is not the correct solution but it is the most suitable 
> > in my opinion.
> > I totally understand what you say and I'm a little sorry that I could still 
> > help in this same fix.
> >
> > This problem is something I would also like to fix because I am using the 
> > RM_WORK_EXCLUDE
> > for quite some time to fix this issue on my distro.
> > I would like to convert the make-mod-scripts to be a native tool but I 
> > haven't had time for that either.
> >
> > Sorry and thank you for all your dedication and help.
> >
> > Jose
> >
> >>
> >> Cheers,
> >>
> >> Richard
> >>
> >>
> >>
> >>
> >>
> >
> >
> > --
> > Best regards,
> >
> > José Quaresma
> >
> >
> >
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180201): 
https://lists.openembedded.org/g/openembedded-core/message/180201
Mute This Topic: 

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Mon, Apr 17, 2023 at 6:31 PM Jose Quaresma  wrote:
>
>
>
> Richard Purdie  escreveu no dia segunda, 
> 17/04/2023 à(s) 20:51:
>>
>> On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
>> > From: Christoph Lauer 
>> >
>> > With rm_work active, external module signing throws an error:
>> > scripts/sign-file: error while loading shared libraries: libcrypto.so.3: 
>> > cannot open shared object file: No such file or directory
>> > Preserve libraries that sign-file script needs during runtime.
>> >
>> > Signed-off-by: Christoph Lauer 
>> > ---
>> >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
>> >  1 file changed, 3 insertions(+)
>> >
>> > diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
>> > b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
>> > index 28e0807d1d..0e24efc597 100644
>> > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
>> > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
>> > @@ -32,3 +32,6 @@ do_configure() {
>> >   -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
>> >   done
>> >  }
>> > +
>> > +# keep native libraries required for module signing
>> > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
>>
>> I'm really reluctant to take this change as it isn't the way
>> dependencies are meant to work.
>>
>> It sounds like something in a shared workdir is depending on something
>> in a recipe workdir and we simply don't support that. Everything needed
>> should be in the shared workdir. At best this is a bandaid and there
>> will be other ways to make this fail such as cleaning make-mod-scripts.
>
>
> The problem is because for signing the kernel modules the sign-file.c [1] is 
> linked dynamically with openssl-native.
> This works when building the in tree kernel modules but will fail when we try 
> to sing any out of tree kernel module.
> To sign the out of tree kernel we will use the binaries from the shared 
> workdir but the native libcrypto.so.3 is removed by
> the rm_work bbclass. We need to link the sign-file statically otherwise it 
> will not work with the rm_work bbclass.
>
> [1] https://github.com/torvalds/linux/blob/master/scripts/sign-file.c
>
> Another solution for this problem can be changing the make-mod-scripts to be 
> a native tool and in this way
> they will be installed and the dependencies will be handled correctly.

There would very likely be different issues if the scripts were
generated and then packaged as a native tool / package. Since they are
so tightly coupled to the kernel. We'd just trade one set of issues
for another (out of sync artifacts, etc).

I'm going to hack on this a bit.

That being said, I've never done any module signing .. since I don't
need it in my development workflow.

Is there a canonical guide to getting it setup so I can test my static
link and relocated artifacts fixes ? is it with meta-integrity and the
kernel-modsign bbclass ?

Bruce


Bruce

>
>>
>> I'm even less keen to take it when I think it's going to be backported
>> "everywhere" as if is the correct solution too.
>>
>> I don't know what the right fix is unfortunately. I'm sure people would
>> like me to think about it and come up with one but there are simply too
>> many different things people would like me to do that with and even for
>> me, it does take a while to work these things out. I'm just out of
>> bandwidth, sorry :(
>
>
> It is true that it is not the correct solution but it is the most suitable in 
> my opinion.
> I totally understand what you say and I'm a little sorry that I could still 
> help in this same fix.
>
> This problem is something I would also like to fix because I am using the 
> RM_WORK_EXCLUDE
> for quite some time to fix this issue on my distro.
> I would like to convert the make-mod-scripts to be a native tool but I 
> haven't had time for that either.
>
> Sorry and thank you for all your dedication and help.
>
> Jose
>
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
>>
>>
>
>
> --
> Best regards,
>
> José Quaresma
>
> 
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180200): 
https://lists.openembedded.org/g/openembedded-core/message/180200
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Richard Purdie  escreveu no dia
segunda, 17/04/2023 à(s) 20:51:

> On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
> > From: Christoph Lauer 
> >
> > With rm_work active, external module signing throws an error:
> > scripts/sign-file: error while loading shared libraries: libcrypto.so.3:
> cannot open shared object file: No such file or directory
> > Preserve libraries that sign-file script needs during runtime.
> >
> > Signed-off-by: Christoph Lauer 
> > ---
> >  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/
> make-mod-scripts_1.0.bb
> > index 28e0807d1d..0e24efc597 100644
> > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > @@ -32,3 +32,6 @@ do_configure() {
> >   -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> >   done
> >  }
> > +
> > +# keep native libraries required for module signing
> > +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
>
> I'm really reluctant to take this change as it isn't the way
> dependencies are meant to work.
>
> It sounds like something in a shared workdir is depending on something
> in a recipe workdir and we simply don't support that. Everything needed
> should be in the shared workdir. At best this is a bandaid and there
> will be other ways to make this fail such as cleaning make-mod-scripts.
>

The problem is because for signing the kernel modules the sign-file.c [1]
is linked dynamically with openssl-native.
This works when building the in tree kernel modules but will fail when we
try to sing any out of tree kernel module.
To sign the out of tree kernel we will use the binaries from the shared
workdir but the native libcrypto.so.3 is removed by
the rm_work bbclass. We need to link the sign-file statically otherwise it
will not work with the rm_work bbclass.

[1] https://github.com/torvalds/linux/blob/master/scripts/sign-file.c

Another solution for this problem can be changing the make-mod-scripts to
be a native tool and in this way
they will be installed and the dependencies will be handled correctly.


> I'm even less keen to take it when I think it's going to be backported
> "everywhere" as if is the correct solution too.
>
> I don't know what the right fix is unfortunately. I'm sure people would
> like me to think about it and come up with one but there are simply too
> many different things people would like me to do that with and even for
> me, it does take a while to work these things out. I'm just out of
> bandwidth, sorry :(
>

It is true that it is not the correct solution but it is the most suitable
in my opinion.
I totally understand what you say and I'm a little sorry that I could still
help in this same fix.

This problem is something I would also like to fix because I am using the
RM_WORK_EXCLUDE
for quite some time to fix this issue on my distro.
I would like to convert the make-mod-scripts to be a native tool but I
haven't had time for that either.

Sorry and thank you for all your dedication and help.

Jose


> Cheers,
>
> Richard
>
>
>
> 
>
>

-- 
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180169): 
https://lists.openembedded.org/g/openembedded-core/message/180169
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Richard Purdie]

On Sun, 2023-04-16 at 12:30 +0200, Christoph Lauer wrote:
> From: Christoph Lauer 
> 
> With rm_work active, external module signing throws an error:
> scripts/sign-file: error while loading shared libraries: libcrypto.so.3: 
> cannot open shared object file: No such file or directory
> Preserve libraries that sign-file script needs during runtime.
> 
> Signed-off-by: Christoph Lauer 
> ---
>  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
> b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> index 28e0807d1d..0e24efc597 100644
> --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> @@ -32,3 +32,6 @@ do_configure() {
>   -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
>   done
>  }
> +
> +# keep native libraries required for module signing
> +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"

I'm really reluctant to take this change as it isn't the way
dependencies are meant to work.

It sounds like something in a shared workdir is depending on something
in a recipe workdir and we simply don't support that. Everything needed
should be in the shared workdir. At best this is a bandaid and there
will be other ways to make this fail such as cleaning make-mod-scripts.

I'm even less keen to take it when I think it's going to be backported
"everywhere" as if is the correct solution too.

I don't know what the right fix is unfortunately. I'm sure people would
like me to think about it and come up with one but there are simply too
many different things people would like me to do that with and even for
me, it does take a while to work these things out. I'm just out of
bandwidth, sorry :(

Cheers,

Richard



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180167): 
https://lists.openembedded.org/g/openembedded-core/message/180167
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Bruce Ashfield]

On Sun, Apr 16, 2023 at 6:31 AM Christoph Lauer
 wrote:
>
> From: Christoph Lauer 
>
> With rm_work active, external module signing throws an error:
> scripts/sign-file: error while loading shared libraries: libcrypto.so.3: 
> cannot open shared object file: No such file or directory
> Preserve libraries that sign-file script needs during runtime.
>

We are by design installing the output of the scripts target to the
staging kernel build dir.

If there's some part of the build and install that isn't going to the
shared location, then we should be making sure it goes to that shared
location (and is cleaned with the rest of the artifacts).

Users of those same scripts need to be able to locate and load the
artifacts from the staging kernel build dir, but that's
solvable/preferable.

If you've tried the above and it doesn't work, it would be useful to
capture that in the commit log.

Bruce

> Signed-off-by: Christoph Lauer 
> ---
>  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
> b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> index 28e0807d1d..0e24efc597 100644
> --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> @@ -32,3 +32,6 @@ do_configure() {
> -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> done
>  }
> +
> +# keep native libraries required for module signing
> +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
> --
> 2.17.1
>
>
> 
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180166): 
https://lists.openembedded.org/g/openembedded-core/message/180166
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Christoph Lauer]

Hi Jose,

Thanks for your comment; I intend to backport the patch to all actively
maintained branches once it is accepted for master merge.

Christoph

Am 17.04.23 um 12:19 schrieb Jose Quaresma:

Hi Christoph,

This patch is also applicable on kirstone so it can be backported.
I use in my distro RM_WORK_EXCLUDE += "make-mod-scripts" but your fix
looks better.
Thanks.

Jose

Christoph Lauer mailto:christoph.la...@email.de>> escreveu no dia domingo, 16/04/2023
à(s) 11:31:

From: Christoph Lauer mailto:christoph.la...@xtronic.de>>

With rm_work active, external module signing throws an error:
scripts/sign-file: error while loading shared libraries:
libcrypto.so.3: cannot open shared object file: No such file or
directory
Preserve libraries that sign-file script needs during runtime.

Signed-off-by: Christoph Lauer mailto:christoph.la...@xtronic.de>>
---
  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
 | 3 +++
  1 file changed, 3 insertions(+)

diff --git
a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb

b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb

index 28e0807d1d..0e24efc597 100644
--- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb

+++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb

@@ -32,3 +32,6 @@ do_configure() {
                 -C ${STAGING_KERNEL_DIR}
O=${STAGING_KERNEL_BUILDDIR} $t
         done
  }
+
+# keep native libraries required for module signing
+RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
--
2.17.1







--
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180165): 
https://lists.openembedded.org/g/openembedded-core/message/180165
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Jose Quaresma]

Hi Christoph,

This patch is also applicable on kirstone so it can be backported.
I use in my distro RM_WORK_EXCLUDE += "make-mod-scripts" but your fix looks
better.
Thanks.

Jose

Christoph Lauer  escreveu no dia domingo,
16/04/2023 à(s) 11:31:

> From: Christoph Lauer 
>
> With rm_work active, external module signing throws an error:
> scripts/sign-file: error while loading shared libraries: libcrypto.so.3:
> cannot open shared object file: No such file or directory
> Preserve libraries that sign-file script needs during runtime.
>
> Signed-off-by: Christoph Lauer 
> ---
>  meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> index 28e0807d1d..0e24efc597 100644
> --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> @@ -32,3 +32,6 @@ do_configure() {
> -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> done
>  }
> +
> +# keep native libraries required for module signing
> +RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
> --
> 2.17.1
>
>
> 
>
>

-- 
Best regards,

José Quaresma

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180157): 
https://lists.openembedded.org/g/openembedded-core/message/180157
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-

[OE-core] [PATCH] make-mod-scripts: preserve libraries when rm_work is used

[Christoph Lauer]

From: Christoph Lauer 

With rm_work active, external module signing throws an error:
scripts/sign-file: error while loading shared libraries: libcrypto.so.3: cannot 
open shared object file: No such file or directory
Preserve libraries that sign-file script needs during runtime.

Signed-off-by: Christoph Lauer 
---
 meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb 
b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
index 28e0807d1d..0e24efc597 100644
--- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
+++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
@@ -32,3 +32,6 @@ do_configure() {
-C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
done
 }
+
+# keep native libraries required for module signing
+RM_WORK_EXCLUDE_ITEMS += "recipe-sysroot-native"
--
2.17.1


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#180113): 
https://lists.openembedded.org/g/openembedded-core/message/180113
Mute This Topic: https://lists.openembedded.org/mt/98296212/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub 
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-