It is no secret that we've needed to formalize some kind of sensible sharing between git repos with common ancestry. The opportunity for sharing the mainline kernel base between linux-yocto and linux-yocto-dev is the main obvious candidate.
Many of us have long been doing local hacks to pre-populate the downloads dir and avoid lengthy/large downloads, and some have even been kind enough to try and get a variant of that upstream[1]. But the problem is that it isn't as simple as adding a new "--reference" switch to the fetcher and then walking away. That will result in non-portable and incompatible use cases with existing work flows. We have to put some thought into how we see this fitting into the larger framework of things and in what ways we want to support use of it. For example, we know we want relative paths in anything in the download dir, but "clone --reference" will put an absolute path in the alternates file by default. We also don't want the SRC_URI to be containing any absolute paths either. We also want (pre)mirror stuff to keep working. So we set some basic ground rules. Firstly we don't even expose the obvious "--reference" argument that some of us are used to at all. That is because we are going to have two types of references - the common one based on objets/info/alternates (this commit) and a pack based one (the subsequent commit). We also restrict any references to be peers of the other content in downloads - no /home/fred/super-SDK/kernel-4.19-rt/ type stuff. The idea is that once you factor out the common building blocks of source, then your kernel (staying with that example) becomes a very small repo. For example - all of the linux-yocto BSPs (standard and -rt) can be held in a repo that is only 20-30 megabytes - small enough to attach to email. Clones via alternates have some limitations - for example there is an arbitrary limit of six[2] levels of nesting, so a pack reference is introduced to resolve this subsequently. However alternaltes make sense for shared non-static (i.e. WIP repos) endpoints, like stable and -rt. [1] https://lists.openembedded.org/g/bitbake-devel/topic/77645766 [2] https://github.com/git/git/commit/c2f493a4 # if (depth > 5) Signed-off-by: Paul Gortmaker <[email protected]> --- .../bitbake-user-manual-fetching.rst | 6 +++++ bitbake/lib/bb/fetch2/git.py | 24 +++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst index 94f4cf3363f6..12a68accc2c5 100644 --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.rst @@ -447,6 +447,12 @@ This fetcher supports the following parameters: parameter implies no branch and only works when the transfer protocol is ``file://``. +- *"altref":* Enables cloned ``git://`` URLs to use the named peer from + the downloads dir as a reference through the standard git alternates + file for clone (see ``-reference``) and subseqent operations. Recipe + is responsible (via fetch dependency) for ensuring the reference is + populated/cloned prior to being called on to act as a reference. + Here are some example URLs: :: SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1" diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py index 6cd5d09b2ded..e742010229c3 100644 --- a/bitbake/lib/bb/fetch2/git.py +++ b/bitbake/lib/bb/fetch2/git.py @@ -59,6 +59,12 @@ Supported SRC_URI options are: Once cloned, nothing will change, so optimize accordingly. The default is "0", set static=1 when appropriate. +- altref + Repository is to use named peer from downloads dir as a reference through + the standard git objects/info/alternates file for clone and subseqent + operations. Recipe is responsible (via fetch dependency) for ensuring the + reference is populated/cloned prior to being called on to act as a reference. + - usehead For local git:// urls to use the current branch HEAD as the revision for use with AUTOREV. Implies nobranch. @@ -165,6 +171,8 @@ class Git(FetchMethod): ud.static = ud.parm.get("static","0") == "1" + ud.altref = ud.parm.get("altref","") + ud.dlname = ud.parm.get("dlname","") # usehead implies nobranch @@ -380,9 +388,20 @@ class Git(FetchMethod): repourl = self._get_repo_url(ud) + refname = ud.altref + if refname: + alts = os.path.join(ud.clonedir, 'objects', 'info', 'alternates') + gitdir = d.getVar("GITDIR") or (d.getVar("DL_DIR") + "/git2") + refdir = os.path.join(gitdir, refname) + if not os.path.exists(refdir): + raise bb.fetch2.FetchError("Unable to find reference %s in %s" % (refname, gitdir)) + # If the repo still doesn't exist, fallback to cloning it if not os.path.exists(ud.clonedir): extcloneargs = d.getVar('GITCLONEARGS_' + ud.names[0]) or d.getVar('GITCLONEARGS') or "--bare --mirror" + if refname: + extcloneargs += " --reference %s" % refdir + # We do this since git will use a "-l" option automatically for local urls where possible if repourl.startswith("file://"): repourl = repourl[7:] @@ -392,6 +411,11 @@ class Git(FetchMethod): progresshandler = GitProgressHandler(d) runfetchcmd(clone_cmd, d, log=progresshandler) + # Fix up reference to be relative for tarball/mirror portability + if ud.altref: + with open(alts, "w") as f: + f.write("../../%s/objects\n" % ud.altref) + if ud.static: runfetchcmd("%s config --bool --add bitbake.static 1" % ud.basecmd, d, workdir=ud.clonedir) -- 2.25.1
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9641): https://lists.yoctoproject.org/g/linux-yocto/message/9641 Mute This Topic: https://lists.yoctoproject.org/mt/81808160/21656 Group Owner: [email protected] Unsubscribe: https://lists.yoctoproject.org/g/linux-yocto/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
