On 10/2/19 6:17 AM, Richard Purdie wrote:
On Fri, 2019-09-27 at 14:56 -0400, Konrad Scherer wrote:
From: Konrad Scherer <[email protected]>
Using the glob function to map signatures to sstate files is very
slow
when the sstate is large and accessed over nfs. The lookup now only
loads the necessary prefixes and doesn't use glob as all.
Unfortunately I don't have access to the systems where the
performance
isse was noticed and on my test system the glob is fast enough that
the performance numbers aren't useful. I could verify that file list
returned by the new code is the same.
[YOCTO #13539]
Signed-off-by: Konrad Scherer <[email protected]>
---
meta/lib/oe/copy_buildsystem.py | 3 ++-
scripts/gen-lockedsig-cache | 44 +++++++++++++++++++++++++++++
----
2 files changed, 41 insertions(+), 6 deletions(-)
Thanks for this, its hugely appreciated!
I aimed to do some profile measurements to show the difference in speed
but when testing it failed with:
Filtering out gdb-cross-x86_64:do_patch
Filtering out gdb-cross-x86_64:do_prepare_recipe_sysroot
Filtering out gdb-cross-x86_64:do_unpack
Gathering file list
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache",
line 77, in <module>
sstate_content_cache[prefix] = build_sha_cache(prefix)
File
"/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache",
line 39, in build_sha_cache
map_sha_to_files(sstate_dir, prefix, sha_map)
File
"/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache",
line 29, in map_sha_to_files
sha = extract_sha(f)
File
"/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache",
line 21, in extract_sha
return filename.split(':')[7].split('_')[0]
IndexError: list index out of range
and then when I fixed that by ignoring files which don't match the pattern:
Gathering file list
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache",
line 80, in <module>
sstate_content_cache[prefix] = build_sha_cache(prefix)
File
"/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache",
line 45, in build_sha_cache
map_sha_to_files(native_sstate_dir, prefix, sha_map)
File
"/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache",
line 27, in map_sha_to_files
sstate_files = os.listdir(sstate_prefix_path)
FileNotFoundError: [Errno 2] No such file or directory:
'/srv/autobuilder/autobuilder.yoctoproject.org/pub/sstateuniversal/b6/'
I therefore added:
diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
index ae5e09d89f..48cb67112f 100755
--- a/scripts/gen-lockedsig-cache
+++ b/scripts/gen-lockedsig-cache
@@ -26,10 +26,13 @@ def map_sha_to_files(dir_, prefix, sha_map):
sstate_prefix_path = dir_ + '/' + prefix + '/'
sstate_files = os.listdir(sstate_prefix_path)
for f in sstate_files:
- sha = extract_sha(f)
- if sha not in sha_map:
- sha_map[sha] = []
- sha_map[sha].append(sstate_prefix_path + f)
+ try:
+ sha = extract_sha(f)
+ if sha not in sha_map:
+ sha_map[sha] = []
+ sha_map[sha].append(sstate_prefix_path + f)
+ except IndexError:
+ continue
# given a prefix build a map of hash to list of files
def build_sha_cache(prefix):
@@ -38,7 +41,7 @@ def build_sha_cache(prefix):
sstate_dir = sys.argv[2]
map_sha_to_files(sstate_dir, prefix, sha_map)
- native_sstate_dir = sys.argv[2] + sys.argv[4]
+ native_sstate_dir = sys.argv[2] + '/' + sys.argv[4]
map_sha_to_files(native_sstate_dir, prefix, sha_map)
return sha_map
My benchmark before was seeing it spend over 30 minutes in bitbake
core-image-minimal:do_populate_sdk_ext on an otherwise idle autobuilder
cluster/NAS (35 minutes from a clean tmpdir).
With the patch applied and my above tweak, I saw:
real 6m58.120s
and I'd note this was with a full build running on the other workers so
the NAS was under load. I could try and get an exact time for the above
but didn't really see the point in spending another 30 minutes on it.
This is the time for the whole SDK image, not just the time this script
takes but its enough for me to say its a vast improvement! :)
Great! It was frustrating to develop the code without a good reproducer.
Konrad: Mind if I squash in the above tweaks?
Not at all. Sorry that you had to spend time debugging my code. That sha
extraction code should have been more robust.
Also, the new code is a bit too chatty and leads to a lot of log
output, we'll need to tweak that too.
Please do.
--
Konrad Scherer, MTS, Linux Products Group, Wind River
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
