On Fri, 2026-07-31 at 09:26 +0000, Jamin Lin wrote: > When a recipe uses multiple git SRC_URI entries with different destsuffix > values (e.g. recipes with separate repositories for the kernel, modules > and application), do_unpack clones each source tree with > 'git clone -n -s'. > > The -s flag uses git's shared-object mechanism: > instead of copying objects locally it writes a .git/objects/info/alternates > file > pointing back to the bare repository under the downloads directory > (DL_DIR/git2/). > > scriptutils.git_convert_standalone_clone() is called by devtool_post_unpack to > make the top-level source directory standalone: it runs 'git repack -a' to > copy > all objects into the local object store and then removes the alternates file. > > However it only processes the top-level source directory. Each nested git repo > created by a separate SRC_URI entry retains its own alternates file still > pointing into downloads/. > > Steps to reproduce: > 1. devtool modify <recipe-with-multiple-git-SRC_URI> > 2. bitbake -c cleanall <recipe> > 3. bitbake <recipe> > > At step 2, 'bitbake -c cleanall' calls fetcher.clean() which deletes > the bare repositories from downloads/git2/. The top-level workspace > repo is standalone (alternates already removed by the original code), > but the nested repos still hold alternates pointing to the now-deleted > paths. > > At step 3, srctree_hash_files() runs 'git add -A .' with a custom > GIT_INDEX_FILE. Git internally calls 'git status --porcelain=2' on > each nested repo to check for changes; this fails with exit 128 because > the nested alternates are broken: > error: unable to normalize alternate object path: > .../downloads/git2/github.com.example.module//objects > fatal: bad object HEAD > fatal: 'git status --porcelain=2' failed in submodule modules/lib/module > > This halts the BitBake parse phase with a CalledProcessError and leaves > the workspace in an unrecoverable state without manual intervention. > > Fix by having devtool_post_unpack() look at the recipe's SRC_URI directly: > any git entry with an explicit destsuffix param names an additional > checkout nested under the source tree, so convert each of those to a > standalone clone the same way as the top-level tree. > > This is deliberately metadata-driven rather than walking the unpacked > source tree looking for '.git' directories: a directory walk has no way > to tell a nested checkout from an ordinary subdirectory of one, so it > either has to stop at the first git repo it finds - which then misses a > destsuffix repo nested inside another repo's own working tree - or keep > walking into every repo's contents, which is wasted work for large trees. > Reading SRC_URI instead gives the exact, authoritative set of paths that > need converting, regardless of how they happen to be nested on disk. > > Only entries with an explicit destsuffix are handled, since that is the > only way a recipe ends up with more than one git checkout under S; this > also avoids having to duplicate the git fetcher's internal logic for > computing an implicit default destsuffix (which depends on the subdir/ > subpath params and BB_GIT_DEFAULT_DESTSUFFIX). > > Signed-off-by: Jamin Lin <[email protected]> > --- > meta/classes/devtool-source.bbclass | 16 ++++++++++++++++ > 1 file changed, 16 insertions(+) > > diff --git a/meta/classes/devtool-source.bbclass > b/meta/classes/devtool-source.bbclass > index f29f40588f..940cdedbac 100644 > --- a/meta/classes/devtool-source.bbclass > +++ b/meta/classes/devtool-source.bbclass > @@ -97,6 +97,22 @@ python devtool_post_unpack() { > > scriptutils.git_convert_standalone_clone(srcsubdir) > > + # Recipes can use multiple git SRC_URI entries with an explicit > destsuffix to > + # unpack several repositories as nested subdirectories of the source tree > + # (e.g. recipes with separate repos for the kernel, modules and > + # application). Each such entry is unpacked as its own 'git clone -s' and > + # needs the same standalone conversion as srcsubdir above, otherwise it > keeps > + # referencing objects in the downloads dir that 'bitbake -c cleanall' > removes. > + # We only look at entries with an explicit destsuffix param, since > that's the > + # only way a recipe ends up with more than one git checkout under S - > this > + # avoids having to duplicate the fetcher's internal default-destsuffix > logic. > + import bb.fetch2 > + fetch = bb.fetch2.Fetch(d.getVar('SRC_URI').split(), d) > + for url in fetch.urls: > + ud = fetch.ud[url] > + if ud.type == 'git' and ud.parm.get('destsuffix'): > + scriptutils.git_convert_standalone_clone(os.path.join(unpackdir, > ud.parm['destsuffix'])) > + > # Make sure that srcsubdir exists > bb.utils.mkdirhier(srcsubdir) > if not os.listdir(srcsubdir):
Why still handle srcsubdir separately above this loop? What about SRC_URI containing multiple git repositories that unpack side-by-side instead of nested? It may be better to just iterate through all git repositories in SRC_URI rather than just the nested ones. Best regards, -- Paul Barker
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#243514): https://lists.openembedded.org/g/openembedded-core/message/243514 Mute This Topic: https://lists.openembedded.org/mt/120531855/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
