On Tue, 2026-06-16 at 15:35 +0200, Anders Heimer via lists.openembedded.org wrote: > Hi Paul, > > On 6/16/26 14:12, Paul Barker wrote: > > On Tue, 2026-06-16 at 10:25 +0200, Anders Heimer wrote: > > > - for pmap in prefixmap: > > > + env = os.environ.copy() > > > + env["LC_ALL"] = "C" > > > + > > > + for pmap, prefix in prefixmap.items(): > > > + dstroot = dvar + prefix > > > # Ignore files from the recipe sysroots (target and native) > > > - cmd = "LC_ALL=C ; sort -z -u '%s' | egrep -v -z > > > '((<internal>|<built-in>)$|/.*recipe-sysroot.*/)' | " % sourcefile > > > + sort_p = subprocess.Popen(["sort", "-z", "-u", "--", > > > sourcefile], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env) > > > + egrep_p = subprocess.Popen(["egrep", "-v", "-z", "-e", > > > r"((<internal>|<built-in>)$|/.*recipe-sysroot.*/)"], stdin=sort_p.stdout, > > > stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env) > > > + sort_p.stdout.close() > > > + > > > # We need to ignore files that are not actually ours > > > # we do this by only paying attention to items from this > > > package > > > - cmd += "fgrep -zw '%s' | " % prefixmap[pmap] > > > + fgrep_p = subprocess.Popen(["fgrep", "-zw", "-e", prefix], > > > stdin=egrep_p.stdout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, > > > env=env) > > > + egrep_p.stdout.close() > > > + > > > # Remove prefix in the source paths > > > - cmd += "sed 's#%s/##g' | " % (prefixmap[pmap]) > > > - cmd += "(cd '%s' ; cpio -pd0mlLu --no-preserve-owner '%s%s' > > > 2>/dev/null)" % (pmap, dvar, prefixmap[pmap]) > > > + sed_p = subprocess.Popen(["sed", "s#%s/##g" % prefix], > > > stdin=fgrep_p.stdout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, > > > env=env) > > > + fgrep_p.stdout.close() > > > + > > > + cpio_p = subprocess.Popen(["cpio", "-pd0mlLu", > > > "--no-preserve-owner", dstroot], stdin=sed_p.stdout, cwd=pmap, > > > stderr=subprocess.DEVNULL, env=env) > > > + sed_p.stdout.close() > > > + > > > + for proc in (cpio_p, sed_p, fgrep_p, egrep_p, sort_p): > > > + proc.wait() > > Hi Anders, thanks for the patches! > > > > If we're reworking this code, I think we should replace the complex > > sed/grep/sort pipeline with Python code. We can read into a Python list > > and sort/filter using the Python standard library, then pass the results > > to cpio. > > Thank you, I am very happy to implement this approach instead. I > strongly agree with all your comments.
Hi Anders, I was discussing this with Richard just now, I may be a bit over eager! This code is performance sensitive as there's sometimes a lot of debug sources to copy. We've seen previously that shutil.copy() is much slower than shelling out to `mv`, so we probably want to keep that one as a subprocess. The rest of the cleanup I suggested is worth doing. Thanks, -- Paul Barker
signature.asc
Description: This is a digitally signed message part
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#238901): https://lists.openembedded.org/g/openembedded-core/message/238901 Mute This Topic: https://lists.openembedded.org/mt/119830169/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
