Hi Paul,
> Subject: Re: [OE-core] [PATCH v1 1/2] externalsrc: Handle nested git repos
> from
> multiple SRC_URI entries
>
> On Tue, 2026-05-19 at 06:28 +0000, Jamin Lin via lists.openembedded.org
> wrote:
> > Hi Paul,
> >
> > > Subject: Re: [PATCH v1 1/2] externalsrc: Handle nested git repos
> > > from multiple SRC_URI entries
> > >
> > > On Fri, 2026-05-15 at 09:36 +0000, Jamin Lin wrote:
> > > > When a recipe uses multiple git SRC_URI entries with different
> > > > destsuffix values (e.g. Zephyr-based recipes with separate repos
> > > > for the kernel, modules, and application), each source is unpacked
> > > > into a subdirectory of EXTERNALSRC that retains its own .git directory.
> > > >
> > > > srctree_hash_files() calls 'git add -A .' at the EXTERNALSRC root,
> > > > which fails with exit code 128 when git encounters these
> > > > unregistered nested git repositories, halting the bitbake parse phase.
> > >
> > > Is this true? The documentation for `git add` [1] talks about
> > > issuing a warning when this occurs, not an error, and in some quick
> > > local testing I get a successful exit (exit code 0) when I try this.
> > >
> >
> > Please try the "zephyr-helloworld" package.
> >
> > This recipe uses multiple Git repositories in "SRC_URI", and I can reproduce
> the issue with this package:
> >
> > https://git.yoctoproject.org/meta-zephyr/tree/meta-zephyr-core/recipes
> > -kernel/zephyr-kernel/zephyr-kernel-src-4.3.0.inc
> >
> > Steps to reproduce:
> >
> > 1. git clone https://github.com/openembedded/bitbake.git
> > 2. git clone https://github.com/openembedded/openembedded-core.git
> > 3. git clone https://git.yoctoproject.org/meta-zephyr
> > 4. git clone https://github.com/openembedded/meta-openembedded.git
> > 5. source openembedded-core/oe-init-build-env build 6. Edit
> > build/conf/bblayers.conf and add the following layers:
> >
> > * meta-openembedded/meta-python
> > * meta-zephyr/meta-zephyr-core
> > * meta-openembedded/meta-oe
> > 7. devtool modify zephyr-helloworld
> > 8. bitbake -c cleanall zephyr-helloworld 9. bitbake zephyr-helloworld
> >
> > After that, the following parser error occurs, and the package can no longer
> be built:
> >
> > ```text
> > jamin_lin@aspeed-fw02:~/oe-review/build$ bitbake zephyr-helloworld
> > Loading cache: 100%
> >
> |###############################################################
> ################################################################
> ################################################################
> ##########| Time: 0:00:01 Loaded 4949 entries from dependency cache.
> > ERROR: ExpansionError during parsing
> > /home/jamin_lin/oe-review/meta-zephyr/meta-zephyr-core/recipes-kernel/
> > zephyr-kernel/zephyr-helloworld.bb
> > bb.data_smart.ExpansionError: Failure expanding variable
> do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which
> triggered exception CalledProcessError: Command '['git', 'add', '-A', '.']'
> returned non-zero exit status 128.
> > The variable dependency chain for the failure is:
> > do_compile[file-checksums]
> >
> > ERROR: Parsing halted due to errors, see error messages above
> > Parsing recipes: 0% |
> | ETA: --:--:--
> > Summary: There were 2 ERROR messages, returning a non-zero exit code.
> > ```
>
> Following these steps I see the same error. But the cause is not the nested
> git
> repositories. `bitbake -c cleanall ...` has deleted the sources from the
> downloads directory, but the devtool workspace is still referencing the
> deleted
> sources. Even a `git status` command in the zephyr-helloworld sources
> directory fails:
>
> $ git -C workspace/sources/zephyr-helloworld status
> error: unable to normalize alternate object path:
> /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
> yrproject-rtos.mcuboot//objects
> error: unable to normalize alternate object path:
> /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
> yrproject-rtos.mcuboot//objects
> error: unable to normalize alternate object path:
> /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
> yrproject-rtos.mcuboot//objects
> error: unable to normalize alternate object path:
> /home/pbarker/bitbake-builds/.bitbake-setup-downloads/git2/github.com.zeph
> yrproject-rtos.mcuboot//objects
> fatal: bad object HEAD
> fatal: 'git status --porcelain=2' failed in submodule bootloader/mcuboot
>
> If I replace step 8 with `bitbake -c cleansstate ...` then there is no parse
> error.
>
> What we need to fix is the interaction between `devtool modify` and `bitbake
> -c cleanall`. Perhaps devtool needs to detect this broken state and tell the
> user
> how to fix it - simply running `bitbake -c fetch zephyr-helloworld` isn't
> possible
> as the metadata can't be parsed.
>
> Best regards,
>
> --
> Paul Barker
Hi Paul,
Thanks for identifying the root cause.
After studying the devtool design, I now understand what happens: when devtool
modify creates the workspace, it does not copy the full git object store
locally.
Instead, it clones the source tree using git's alternate object store mechanism
- the workspace's .git/objects/info/alternates file points back to the
downloaded bare repositories under the downloads/ directory.
This way the workspace shares the git objects with the download cache rather
than duplicating them on disk.
When bitbake -c cleanall is run, it deletes the downloaded bare repositories
from downloads/. The workspace source tree still exists, but its alternates now
point to missing paths.
Any git operation on the workspace (including git status, git add, etc.) then
fails because git cannot resolve the object references, producing errors like:
error: unable to normalize alternate object path: .../downloads/git2/.../objects
fatal: bad object HEAD
This is what causes the parser to halt with exit 128.
The fix on my side is straightforward: replace bitbake -c cleanall with bitbake
-c cleansstate in my devtool workflow. cleansstate clears the sstate cache
without touching the downloads directory, so the workspace alternates remain
intact.
That said, I am not sure whether some improvement on the OE side would still be
worthwhile. Currently, when this breakage occurs, the user sees only a cryptic
CalledProcessError: exit 128 with no indication of what went wrong or how to
recover.
The situation is also not obvious: running bitbake -c cleanall on a single-repo
recipe with a devtool workspace does not trigger this problem, so users working
with multi-repo recipes (such as Zephyr-based recipes that fetch multiple git
repositories into separate destsuffix directories) are likely to hit this
unexpectedly. A clearer error message pointing to the root cause and the
recovery steps (devtool reset -n followed by devtool modify) might save other
users from the same confusion.
Please drop this patch series because it is not a normal test case for devtool.
Below patch that may help improve the user experience, though I leave it to the
community to decide whether they are worth reviewing:
Best regards,
Jamin
diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 902ff2604f..3e6c9d937e 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -227,6 +227,57 @@ def srctree_hash_files(d, srcdir=None):
ret = " "
if git_dir is not None:
+ # Find nested git repos created by multiple SRC_URI git entries with
+ # different destsuffix values, so their alternates can also be checked
+ # for breakage below.
+ nested_git_dirs = []
+ for root, dirs, files in os.walk(s_dir):
+ if root == s_dir:
+ continue
+ if '.git' in dirs or '.git' in files:
+ nested_git_dirs.append(root)
+ dirs[:] = []
+
+ # Check for broken git alternates in the top-level repo and all nested
+ # repos. This can happen when 'bitbake -c cleanall' deletes the
+ # downloads directory that the devtool workspace references via
+ # alternates, causing all subsequent git operations to exit 128 and
+ # halt the parse phase. Relative alternate paths are resolved against
+ # the repo's objects directory; double-slash paths (a known devtool
+ # artefact) are normalised before the existence check.
+ def has_broken_alternates(dot_git):
+ obj_dir = os.path.join(dot_git, 'objects')
+ alt_file = os.path.join(obj_dir, 'info', 'alternates')
+ if not os.path.exists(alt_file):
+ return False
+ with open(alt_file) as f:
+ for line in f:
+ path = line.strip()
+ if not path or path.startswith('#'):
+ continue
+ if not os.path.isabs(path):
+ path = os.path.join(obj_dir, path)
+ if not os.path.exists(os.path.normpath(path)):
+ return True
+ return False
+
+ broken = has_broken_alternates(git_dir)
+ if not broken:
+ for nested in nested_git_dirs:
+ nested_dot_git = os.path.join(nested, '.git')
+ if os.path.isdir(nested_dot_git) and
has_broken_alternates(nested_dot_git):
+ broken = True
+ break
+
+ if broken:
+ bb.warn('%s: devtool workspace has broken git alternates, '
+ 'likely caused by "bitbake -c cleanall" removing the '
+ 'downloads directory. '
+ 'To repair the workspace, run: '
+ 'devtool reset %s && devtool modify %s'
+ % (d.getVar('PN'), d.getVar('PN'), d.getVar('PN')))
+ return s_dir + '/*:True'
+
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#237380):
https://lists.openembedded.org/g/openembedded-core/message/237380
Mute This Topic: https://lists.openembedded.org/mt/119327122/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
