Is it possible to transition from dumping source artifacts and patches directly into $WORKDIR to having a clean set of sub-directories for SRC_URI items? Not having to set S separately for git recipes would be nice too.
Alex On Mon, 30 May 2022 at 12:51, Richard Purdie <[email protected]> wrote: > > I've gone around in circles on this for a long time. The background > isn't obvious at first glance. > > "S" represents where our source is. There is code like do_unpack which > wants to clean S before rerunning the task. It can't just clean > everything in WORKDIR as there is T, recipe-sysroot* and more that need > to be left alone (e.g. recipe-sysroot-native may have > do_fetch/do_unpack dependencies). > > S is usually defined as ${BPN}-${PV} via ${BP}. > > Many SRC_URIs are tarballs and tarballs usually extract into a > directory, which often happens to be BPN-PV. The tarball itself > therefore creates the ${S} directory. > > For git, the git fetcher extracts into git/ so git recipes end up > setting: > > S = "${WORKDIR}/git" > > This was done historically since using ${PV} for the source dir for a > git recipe was likely going to complicate the paths needlessly. > > The problem comes with recipes like keymaps which only have local files > referenced in SRC_URI. These do: > > S = "${WORKDIR}" > > so if do_unpack cleans S, it cleans WORKDIR which is very bad. We have > horrible code to guard against that. This is the first thing it would > be nice to fix. > > We also have a second problem which is that if do_fetch/do_unpack > rerun, a file installed by a SRC_URI in the first run but removed in > the second, will remain in ${WORKDIR}. This means you can change a > recipe and the rebuild might not be deterministic. Again, this is > rather horrible. > > It is very hard for the code to know which files it should be removing > when it reruns do_unpack. This is the second issue I'd ideally like to > fix. > > Ideas: > > The first issue can be improved with code like: > > diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass > index bdb3ac33c67..0da22b34e8a 100644 > --- a/meta/classes/base.bbclass > +++ b/meta/classes/base.bbclass > @@ -161,8 +161,10 @@ python base_do_fetch() { > bb.fatal("Bitbake Fetcher Error: " + repr(e)) > } > > +UNPACK_DIR = "${WORKDIR}" > + > addtask unpack after do_fetch > -do_unpack[dirs] = "${WORKDIR}" > +do_unpack[dirs] = "${UNPACK_DIR}" > > do_unpack[cleandirs] = "${@d.getVar('S') if os.path.normpath(d.getVar('S')) > != os.path.normpath(d.getVar('WORKDIR')) else os.path.join('${S}', > 'patches')}" > > @@ -173,7 +175,7 @@ python base_do_unpack() { > > try: > fetcher = bb.fetch2.Fetch(src_uri, d) > - fetcher.unpack(d.getVar('WORKDIR')) > + fetcher.unpack(d.getVar('UNPACK_DIR')) > except bb.fetch2.BBFetchException as e: > bb.fatal("Bitbake Fetcher Error: " + repr(e)) > } > > > i.e. we make UNPACK_DIR configurable, then a recipe like keymaps can > do: > > diff --git a/meta/recipes-bsp/keymaps/keymaps_1.0.bb > b/meta/recipes-bsp/keymaps/keymaps_1.0.bb > index 84d09cb965d..2927e7f1392 100644 > --- a/meta/recipes-bsp/keymaps/keymaps_1.0.bb > +++ b/meta/recipes-bsp/keymaps/keymaps_1.0.bb > @@ -25,14 +25,14 @@ SRC_URI = "file://keymap.sh" > INITSCRIPT_NAME = "keymap.sh" > INITSCRIPT_PARAMS = "start 01 S ." > > -S = "${WORKDIR}" > +UNPACK_DIR = "${S}" > > do_install () { > # Only install the script if 'sysvinit' is in DISTRO_FEATURES > # THe ulitity this script provides could be achieved by > systemd-vconsole-setup.service > if ${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; > then > install -d ${D}${sysconfdir}/init.d/ > - install -m 0755 ${WORKDIR}/keymap.sh ${D}${sysconfdir}/init.d/ > + install -m 0755 ${S}/keymap.sh ${D}${sysconfdir}/init.d/ > fi > } > > > which I think then has the recipe make more logical sense. > > For the second issue, the best idea I could come up with was to make > do_unpack extract to a subdirectory or WORKDIR, then symlink the files > into place in WORKDIR. do_unpack can then remove any symlink it finds > in WORKDIR when it reruns. > > Whether either or both of these improvements are worthwhile and whether > there is the communuity demand and support to implement them and > migrate metadata to match, I don't know but I wanted to put the ideas > down here. > > Cheers, > > Richard > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#1554): https://lists.openembedded.org/g/openembedded-architecture/message/1554 Mute This Topic: https://lists.openembedded.org/mt/91428199/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-architecture/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
