On Tue, 2026-06-16 at 15:35 +0200, Anders Heimer via lists.openembedded.org wrote: > On 6/16/26 14:12, Paul Barker wrote: > > On Tue, 2026-06-16 at 10:25 +0200, Anders Heimer wrote: > > > - for pmap in prefixmap: > > > + env = os.environ.copy() > > > + env["LC_ALL"] = "C" > > > + > > > + for pmap, prefix in prefixmap.items(): > > > + dstroot = dvar + prefix > > > # Ignore files from the recipe sysroots (target and native) > > > - cmd = "LC_ALL=C ; sort -z -u '%s' | egrep -v -z > > > '((<internal>|<built-in>)$|/.*recipe-sysroot.*/)' | " % sourcefile > > > + sort_p = subprocess.Popen(["sort", "-z", "-u", "--", > > > sourcefile], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env) > > > + egrep_p = subprocess.Popen(["egrep", "-v", "-z", "-e", > > > r"((<internal>|<built-in>)$|/.*recipe-sysroot.*/)"], stdin=sort_p.stdout, > > > stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env) > > > + sort_p.stdout.close() > > > + > > > # We need to ignore files that are not actually ours > > > # we do this by only paying attention to items from this > > > package > > > - cmd += "fgrep -zw '%s' | " % prefixmap[pmap] > > > + fgrep_p = subprocess.Popen(["fgrep", "-zw", "-e", prefix], > > > stdin=egrep_p.stdout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, > > > env=env) > > > + egrep_p.stdout.close() > > > + > > > # Remove prefix in the source paths > > > - cmd += "sed 's#%s/##g' | " % (prefixmap[pmap]) > > > - cmd += "(cd '%s' ; cpio -pd0mlLu --no-preserve-owner '%s%s' > > > 2>/dev/null)" % (pmap, dvar, prefixmap[pmap]) > > > + sed_p = subprocess.Popen(["sed", "s#%s/##g" % prefix], > > > stdin=fgrep_p.stdout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, > > > env=env) > > > + fgrep_p.stdout.close() > > > + > > > + cpio_p = subprocess.Popen(["cpio", "-pd0mlLu", > > > "--no-preserve-owner", dstroot], stdin=sed_p.stdout, cwd=pmap, > > > stderr=subprocess.DEVNULL, env=env) > > > + sed_p.stdout.close() > > > + > > > + for proc in (cpio_p, sed_p, fgrep_p, egrep_p, sort_p): > > > + proc.wait() > > Hi Anders, thanks for the patches! > > > > If we're reworking this code, I think we should replace the complex > > sed/grep/sort pipeline with Python code. We can read into a Python list > > and sort/filter using the Python standard library, then pass the results > > to cpio. > > Thank you, I am very happy to implement this approach instead. I > strongly agree with all your comments.
There are some other things to consider here. This is a fairly sensitive area of code from a performance perspective. You could code much of this in python using shutil for example however shutil has traditionally been up to an order of magnitude slower. As such, this code was optimized to be fast, hence the use of cpio. In many cases I worry less about performance but this is one area it does really matter and makes a big difference to build speed overall. python can be fast if carefully written but if this used shutil for example, it likely won't be. Cheers, Richard
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#238902): https://lists.openembedded.org/g/openembedded-core/message/238902 Mute This Topic: https://lists.openembedded.org/mt/119830169/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
