On the same machine, I had two identical poky directories. One, I built first
to use as the cache source. I then configured the second one to use a
SSTATE_MIRRORS pointing to the first.
The build of the second would just detect changes and force a rebuild. Running
the bitbake-diffsigs tool on the do_fetch...sigdata showed something like the
following on all of the local files:
Removed("/home/user/poky1/meta/recipes...file1", chksum1)
Added("/home/user/poky2/meta/recipes...file1", chksum1)
Upon applying my patch, the components of core-minimal-image were pulled from
the sstate-cache.
I will have to see about the other local file cases that I don't know about. :)
-Jate
________________________________________
From: Paul Eggleton [paul.eggle...@linux.intel.com]
Sent: Wednesday, June 19, 2013 11:45 AM
To: Jate Sujjavanich
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH] Checksums for local files now stored using
partial recipe path
On Wednesday 19 June 2013 16:24:53 Paul Eggleton wrote:
> Hi Jate,
>
> On Wednesday 19 June 2013 11:08:10 Jate Sujjavanich wrote:
> > This allows sstate-cache to be shared between builds in different
> > directories.
> >
> > Differences in the full path were triggering a false positive when there
> > were actually no changes.
> >
> > Signed-off-by: Jate Sujjavanich <jate.sujjavan...@myfuelmaster.com>
> > ---
> >
> > bitbake/lib/bb/fetch2/__init__.py | 14 +++++++++-----
> > bitbake/lib/bb/siggen.py | 3 ++-
> > 2 files changed, 11 insertions(+), 6 deletions(-)
> >
> > diff --git a/bitbake/lib/bb/fetch2/__init__.py
> > b/bitbake/lib/bb/fetch2/__init__.py index dd1cc93..7ab44d7 100644
> > --- a/bitbake/lib/bb/fetch2/__init__.py
> > +++ b/bitbake/lib/bb/fetch2/__init__.py
> >
> > @@ -900,8 +900,7 @@ def get_checksum_file_list(d):
> > return " ".join(filelist)
> >
> > -
> > -def get_file_checksums(filelist, pn):
> >
> > +def get_file_checksums(filelist, pn, topdir):
> > """Get a list of the checksums for a list of local files
> >
> > Returns the checksums for a list of local files, caching the results
> > as
> >
> > @@ -917,7 +916,12 @@ def get_file_checksums(filelist, pn): bb.warn("Unable
> > to get checksum for %s SRC_URI entry %s: %s" % (pn, os.path.basename(f),
> > e)) return None
> >
> > return checksum
> >
> > +
> > + (recipe_root, _) = os.path.split(topdir)
> >
> > + def remove_recipe_parent(data):
> > + return data.replace(recipe_root, '').strip('/')
> > +
> >
> > checksums = []
> >
> > for pth in filelist.split():
> > checksum = None
> >
> > @@ -927,7 +931,7 @@ def get_file_checksums(filelist, pn):
> > for f in glob.glob(pth):
> > checksum = checksum_file(f)
> >
> > if checksum:
> > - checksums.append((f, checksum))
> > + checksums.append((remove_recipe_parent(f),
> > + checksum))
> >
> > elif os.path.isdir(pth):
> > # Handle directories
> >
> > for root, dirs, files in os.walk(pth):
> > @@ -935,12 +939,12 @@ def get_file_checksums(filelist, pn):
> > fullpth = os.path.join(root, name)
> > checksum = checksum_file(fullpth)
> >
> > if checksum:
> > - checksums.append((fullpth, checksum))
> > +
> > + checksums.append((remove_recipe_parent(fullpth), checksum))
> >
> > else:
> > checksum = checksum_file(pth)
> >
> > if checksum:
> > - checksums.append((pth, checksum))
> > + checksums.append((remove_recipe_parent(pth), checksum))
> >
> > checksums.sort(key=operator.itemgetter(1))
> > return checksums
> >
> > diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index
> > 8861337..c64acfe 100644 --- a/bitbake/lib/bb/siggen.py
> > +++ b/bitbake/lib/bb/siggen.py
> >
> > @@ -74,6 +74,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> > self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
> > self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST",
> >
> > True) or "").split()) self.taskwhitelist = None
> > + self.topdir = data.getVar("TOPDIR", True)
> >
> > self.init_rundepcheck(data)
> >
> > def init_rundepcheck(self, data):
> > @@ -187,7 +188,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> > self.runtaskdeps[k].append(dep)
> >
> > if task in dataCache.file_checksums[fn]:
> > - checksums =
> > bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> > recipename) + checksums =
> > + bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> > + recipename, self.topdir)
> >
> > for (f,cs) in checksums:
> > self.file_checksum_values[k][f] = cs
> > data = data + cs
>
> Good catch! The only thing is, this will not help for files within different
> layers which may not be underneath TOPDIR; I think we'll need a function
> that determines which layer the file is under (longest path match from
> data.getVar('BBLAYERS', True).split()) and then take that path off the
> beginning.
>
> Additionally, this is a patch against bitbake so it will need to go to the
> bitbake-de...@lists.openembedded.org mailing list.
Actually, looking more closely at this I'm not sure how the full path to the
file would be getting into the signature - looking at lib/bb/siggen.py it
should only be adding the file checksum value to the signature data and not the
path. I did a quick test with master by moving some files referred to in
SRC_URI to a different valid location (thus changing their full path), cleaning
the recipe and then building it again, and the output was restored from sstate
rather than rebuilding.
Can you explain how you came to the conclusion that this was why the checksums
were different on different machines?
Cheers,
Paul
--
Paul Eggleton
Intel Open Source Technology Centre
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core
