remove_by_stamps() collects the hashes still in use by globbing the stamps
directory at a fixed depth, "*/*/*.do_*", matching <arch>/<pn>/<file>; the
fixed depth is deliberate, so that the walk does not stat() the whole tree,
which over NFS costs a round trip per entry. But the shared-source recipes -
gcc-source, llvm-project-source and rust-source - set STAMP to
${STAMPS_DIR}/work-shared/<pn>-<pv>-<pr>, one level shallower, so their
hashes are never collected and --stamps-dir plans their sstate objects for
removal while a build still uses them. (externalsrc's work-shared/<pn>/ is
at the usual depth and was already found.)
Glob that layout too, at its own fixed depth, rather than recursively.
Against a cache shared by eight build directories, the stock script
planned 740 of 91486 files for removal; 43 of them, all gcc-source,
llvm-project-source and rust-source objects, had hashes in live
work-shared stamps. With this change it plans 697, the same list as a
recursive "**" glob produces. Filesystem calls during the run (strace):
353084 before, 353196 after, against 377568 for the recursive glob.
The shell script this replaced used "find $stamps -maxdepth 3", which did
include them.
Fixes: 2fa1b25d7485 ("sstate-cache-management: Rewrite in python")
AI-Generated: Claude Opus 5.5
Signed-off-by: Alex Kiernan <[email protected]>
---
scripts/sstate-cache-management.py | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/scripts/sstate-cache-management.py
b/scripts/sstate-cache-management.py
index f5737004c544..d088878a92df 100755
--- a/scripts/sstate-cache-management.py
+++ b/scripts/sstate-cache-management.py
@@ -142,6 +142,15 @@ def collect_sstate_paths(args):
return paths
+# Where stamps live under STAMPS_DIR, each at a fixed depth: globbing those
+# depths exactly, rather than recursively, keeps the walk from stat()ing
+# everything in the tree, which over NFS is slow with a round trip per entry.
+STAMP_LAYOUTS = (
+ "*/*/", # <arch>/<pn>/<pv>... (bitbake.conf); also externalsrc's
work-shared/<pn>/
+ "work-shared/", # work-shared/<pn>-<pv>-<pr>... (gcc-source,
llvm-project-source, rust-source)
+)
+
+
def remove_by_stamps(args, paths):
all_sums = set()
for stamps_dir in args.stamps_dir:
@@ -151,14 +160,16 @@ def remove_by_stamps(args, paths):
all_sums |= set(
[
re_sigdata.search(x.parts[-1]).group(1)
- for x in stamps_path.glob("*/*/*.do_*.sigdata.*")
+ for layout in STAMP_LAYOUTS
+ for x in stamps_path.glob(layout + "*.do_*.sigdata.*")
]
)
re_setscene = re.compile(r"do_.*_setscene\.([^.]*)")
all_sums |= set(
[
re_setscene.search(x.parts[-1]).group(1)
- for x in stamps_path.glob("*/*/*.do_*_setscene.*")
+ for layout in STAMP_LAYOUTS
+ for x in stamps_path.glob(layout + "*.do_*_setscene.*")
]
)
return [p for p in paths if p.bb_unihash not in all_sums]
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#247217):
https://lists.openembedded.org/g/openembedded-core/message/247217
Mute This Topic: https://lists.openembedded.org/mt/121587341/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
