On Mon, 2020-03-23 at 17:30 +0000, Paul Barker wrote:
> On Thu, 5 Mar 2020 22:56:15 +0000
> "Freihofer, Adrian" <adrian.freiho...@siemens.com> wrote:
>
> > On Thu, 2020-03-05 at 09:42 +0000, Paul Barker wrote:
> > > On Wed, 4 Mar 2020 12:30:10 +0000
> > > "Freihofer, Adrian" <adrian.freiho...@siemens.com> wrote:
> > >
> > > > -----Original Message-----
> > > > From: Paul Barker <pbar...@konsulko.com>
> > > > To: "Freihofer, Adrian" <adrian.freiho...@siemens.com>
> > > > Cc: openembedded-core@lists.openembedded.org <
> > > > openembedded-core@lists.openembedded.org>
> > > > Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> > > > Date: Wed, 04 Mar 2020 09:59:44 +0000
> > > >
> > > > On Wed, 4 Mar 2020 08:12:27 +0000
> > > > "Freihofer, Adrian" <adrian.freiho...@siemens.com> wrote:
> > > >
> > > > > The unpack function failed because the submodules were not
> > > > > downloaded.
> > > > > Calling download before unpack for each submodule solves this
> > > > > issue.
> > > > >
> > > > > Signed-off-by: Adrian Freihofer <adrian.freiho...@siemens.com>
> > > > > ---
> > > > > bitbake/lib/bb/fetch2/gitsm.py | 1 +
> > > > > 1 file changed, 1 insertion(+)
> > > > >
> > > > > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > > > > b/bitbake/lib/bb/fetch2/gitsm.py
> > > > > index c622771d21..3715e9824f 100644
> > > > > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > > > > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > > > > @@ -184,6 +184,7 @@ class GitSM(Git):
> > > > >
> > > > > try:
> > > > > newfetch = Fetch([url], d, cache=False)
> > > > > + newfetch.download()
> > > > > newfetch.unpack(root=os.path.dirname(os.path.joi
> > > > > n(re
> > > > > po
> > > > > _conf, 'modules', module)))
> > > > > except Exception as e:
> > > > > logger.error('gitsm: submodule unpack failed: %s
> > > > > %s'
> > > > > %
> > > > > (type(e).__name__, str(e)))
> > > >
> > > > You shouldn't be trying to download submodules in the do_unpack
> > > > step.
> > > > If
> > > > they're missing at this stage it probably indicates a fetch issue.
> > > >
> > > > Basically true, but the information about which submodules need to
> > > > be
> > > > downloaded is not available before the top level repo has been
> > > > unpacked. I guess the solution must be something like:
> > > >
> > > > fetch top-level
> > > > unpack top-level
> > > > foreach submodule:
> > > > fetch submodule
> > > > unpack submodule
> > > >
> > > > What's the exact error that you're seeing?
> > > >
> > > > Don't remeber exactly. But when I debugged the flow, it became
> > > > obvious,
> > > > that the submodules are not downloaded.
> > > >
> > > > This could be related to the issue I saw when the fetcher uses git
> > > > shallow
> > > > tarballs from a mirror - if that's the case I'm planning to get
> > > > that
> > > > fixed
> > > > this weekend.
> > > >
> > > > Yes, the problem occurs with git shallows. The patch solves it with
> > > > and
> > > > without shallows for us.
> > > >
> > > > Also, your email client seems to have chewed the patch up. It's
> > > > best to
> > > > send
> > > > patches using `git send-email` only.
> > > >
> > > > Sorry, we switched to a new e-mail solution. I have to adapt my
> > > > setup.
> > >
> > > The problem with your patch is that it would require network access
> > > during
> > > do_unpack which then breaks our ability to do offline builds.
> > Fetching in the unpack task is probably really wrong, I agree.
> > However, it also works on our CI infrastructure which runs without
> > Internet connectivity.
> > > Do you only see this issue when fetching from git shallow mirror
> > > tarballs?
> >
> > On our CI which works with gitshallow archives, fetching did not work
> > at all. On machines with Internet access we also had problems, but I did
> > not completely analyze what happened. The ovmf package from poky was
> > causing the troubles in our case.
> >
> > > If so, the mirror tarballs need to be extracted during do_fetch but
> > > not placed in the usual 'git2' directory (as future fetches can't
> > > cope with shallow clones in that directory).
> >
> > Do you already have an idea how this could be implemented?
>
> Sorry for the late response, I meant to let you know when my fix for this had
> gone in to bitbake but I ended up away due to illness for a couple of weeks.
> Commit 3987db95 "fetch2/gitsm: Unpack shallow mirror tarballs" should resolve
> your issue, please let me know if it doesn't.
Thank you Paul. We picked your commit during the last days. It looks
much better than mine and it works perfect.
Regards,
Adrian
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#136673):
https://lists.openembedded.org/g/openembedded-core/message/136673
Mute This Topic: https://lists.openembedded.org/mt/72395274/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
