On Wed, May 22, 2024 at 3:09 PM Richard Purdie via
lists.openembedded.org
<richard.purdie=linuxfoundation....@lists.openembedded.org> wrote:
>
> On Wed, 2024-05-22 at 13:00 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: openembedded-core@lists.openembedded.org
> > > <openembedded-core@lists.openembedded.org> On Behalf Of Richard Purdie
> > > Sent: den 22 maj 2024 11:28
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [PATCH 1/3] base: Switch UNPACKDIR to a subdir of
> > > WORKDIR
> > >
> > > Change do_unpack to unpack files to a subdirectory of WORKDIR instead of
> > > WORKDIR
> > > itself. There are several good reasons for this but it is mainly about
> > > being able
> > > to isolate the output of the unpack task and tell the files apart from
> > > other things
> > > which are created in workdir (logs, sysroots, temp dirs and more).
> > >
> > > This means that when the do_unpack task reruns, we can clean UNPACKDIR
> > > and know
> > > we have a standard point to start builds from.
> > >
> > > It also makes code in tools like devtool and recipetool easier.
> > >
> > > To reduce the impact to users, if a subdirectory under UNPACKDIR matches
> > > the first subdirectory under WORKDIR of S, that directory is moved into
> > > position
> > > inside WORKDIR. This preserves the behaviour of S = "${WORKDIR}/git",
> > > S = "${WORKDIR}/${BPN}" and other commonly used source directory setups.
> > >
> > > The directory is moved since sadly many autotools based projects can't
> > > cope with
> > > symlinks in their paths.
> > >
> > > The patch also updates reproducible and SOURCE_DATE_EPOCH handling to
> > > match the new potential source locations. We can get rid of the horrible
> > > list of hardcoded directories in WORKDIR to ignore from that code.
> > >
> > > Signed-off-by: Richard Purdie <richard.pur...@linuxfoundation.org>
> > > ---
> > > meta/classes-global/base.bbclass | 28 +++++++++++++++++++++++-----
> > > meta/conf/bitbake.conf | 2 +-
> > > meta/lib/oe/reproducible.py | 19 ++++++++++---------
> > > 3 files changed, 34 insertions(+), 15 deletions(-)
> > >
> > > diff --git a/meta/classes-global/base.bbclass
> > > b/meta/classes-global/base.bbclass
> > > index 066f3848f7c..13e91b24a3b 100644
> > > --- a/meta/classes-global/base.bbclass
> > > +++ b/meta/classes-global/base.bbclass
> > > @@ -153,20 +153,38 @@ python base_do_fetch() {
> > > }
> > >
> > > addtask unpack after do_fetch
> > > -do_unpack[dirs] = "${UNPACKDIR}"
> > > -
> > > -do_unpack[cleandirs] = "${@d.getVar('S') if
> > > os.path.normpath(d.getVar('S')) != os.path.normpath(d.getVar('WORKDIR'))
> > > else os.path.join('${S}', 'patches')}"
> > > +do_unpack[cleandirs] = "${UNPACKDIR}"
> > >
> > > python base_do_unpack() {
> > > + import shutil
> > > +
> > > + sourcedir = d.getVar('S')
> > > + # Intentionally keep SOURCE_BASEDIR internal to the task just for SDE
> > > + d.setVar("SOURCE_BASEDIR", sourcedir)
> > > +
> > > src_uri = (d.getVar('SRC_URI') or "").split()
> > > if not src_uri:
> > > return
> > >
> > > + basedir = None
> > > + unpackdir = d.getVar('UNPACKDIR')
> > > + workdir = d.getVar('WORKDIR')
> > > + if sourcedir.startswith(workdir) and not
> > > sourcedir.startswith(unpackdir):
> > > + basedir = sourcedir.replace(workdir, '').strip("/").split('/')[0]
> > > + if basedir:
> > > + bb.utils.remove(workdir + '/' + basedir, True)
> > > + d.setVar("SOURCE_BASEDIR", workdir + '/' + basedir)
> > > +
> > > try:
> > > fetcher = bb.fetch2.Fetch(src_uri, d)
> > > fetcher.unpack(d.getVar('UNPACKDIR'))
> > > except bb.fetch2.BBFetchException as e:
> > > bb.fatal("Bitbake Fetcher Error: " + repr(e))
> > > +
> > > + if basedir and os.path.exists(unpackdir + '/' + basedir):
> > > + # Compatibility magic to ensure ${WORKDIR}/git and
> > > ${WORKDIR}/${BP}
> > > + # as often used in S work as expected.
> > > + shutil.move(unpackdir + '/' + basedir, workdir + '/' + basedir)
> > > }
> > >
> > > SSTATETASKS += "do_deploy_source_date_epoch"
> > > @@ -199,8 +217,8 @@ addtask do_deploy_source_date_epoch_setscene
> > > addtask do_deploy_source_date_epoch before do_configure after do_patch
> > >
> > > python create_source_date_epoch_stamp() {
> > > - # Version: 1
> > > - source_date_epoch = oe.reproducible.get_source_date_epoch(d,
> > > d.getVar('S'))
> > > + # Version: 2
> > > + source_date_epoch = oe.reproducible.get_source_date_epoch(d,
> > > d.getVar('SOURCE_BASEDIR') or d.getVar('S'))
> > > oe.reproducible.epochfile_write(source_date_epoch,
> > > d.getVar('SDE_FILE'), d)
> > > }
> > > do_unpack[postfuncs] += "create_source_date_epoch_stamp"
> >
> > The following code is part of the anonymous python() function in
> > base.bbclass:
> >
> > if os.path.normpath(d.getVar("WORKDIR")) !=
> > os.path.normpath(d.getVar("S")):
> > d.appendVar("PSEUDO_IGNORE_PATHS", ",${S}")
> > if os.path.normpath(d.getVar("WORKDIR")) !=
> > os.path.normpath(d.getVar("B")):
> > d.appendVar("PSEUDO_IGNORE_PATHS", ",${B}")
> >
> > Since it is (or will be) an error to have S = "${WORKDIR}", it should now be
> > possible to move the addition of ${S} to PSEUDO_IGNORE_PATHS to
> > bitbake.conf.
>
> Yes, we should be able to start finding and cleaning up things like
> this.
>
> > I also believe ${UNPACKDIR} should be added to PSEUDO_IGNORE_PATHS.
>
> Why? I don't think we want UNPACKDIR there, we only need S.
There were recently couple issues with pseudo when the source file
from something not unpacked under ${S} confuses pseudo when it sees
the same file unpacked by do_unpack and then linked from
package/usr/src/debug e.g.:
path mismatch [3 links]: ino 214373575 db
'opencv/4.5.5-r0/package/usr/src/debug/lib32-opencv/4.5.5-r0/contrib/modules/intensity_transform/src/bimef.cpp'
req 'opencv/4.5.5-r0/contrib/modules/intensity_transform/src/bimef.cpp'.
which can be usually easily fixed by moving the additional files (or
other repositories) to be unpacked/checked out as subdirectory under
${S} to make them ignored by pseudo as well.
As in:
https://git.openembedded.org/meta-openembedded/commit/?h=kirkstone&id=da98a75f37830e69ded0207cc6c73182ab00ec50
which moved ${WORKDIR}/contrib to ${WORKDIR}/git/contrib to avoid this issue.
But now with clearer separation thanks to ${UNPACKDIR} this whouldn't
be needed if we add whole UNPACKDIR to PSEUDO_IGNORE_PATHS.
Is there any case when UNPACKDIR needs to be tracked by pseudo?
> > And on a related note, would it make sense to make B = ${WORKDIR} an error
> > too?
> > In which case the addition of ${B} to PSEUDO_IGNORE_PATHS could also be
> > moved
> > to bitbake.conf.
>
> I'd hope no one is doing that so yes, I'd also like to move that. We
> can make that an error too.
>
> Cheers,
>
> Richard
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#199751):
https://lists.openembedded.org/g/openembedded-core/message/199751
Mute This Topic: https://lists.openembedded.org/mt/106239726/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
