On Tue, 2013-03-12 at 11:25 +0100, Enrico Scholz wrote: > Richard Purdie <[email protected]> writes: > > >> Old implementation suffered from lot of problems; e.g. > >> > >> * redundant code > >> > >> * calls 'os.stat()' which references files on host; this can give wrong > >> results about existing/non-existing and can cause EPERM (instead of > >> the catched ENONENT) exceptions > >> > >> * does not deal with special cases like '..' leaving the sysroot. > > > > Whilst these changes are good, they do come at a cost: > > > > post symlink package changes > > ./perfscript -c 8c22531e491e6b0cfffaaa80d6bc75db757fc1d1 > > 49:38.46,17:12.15 > > > > pre symlink package changes > > ./perfscript -c 1a80329b3fcf23ecc23e409a260b9b2182652f65 > > 48:16.33,13:39.97 > > > > So it added 1m20 to the overall build time, but more worryingly, added > > added nearly 3m30 to the time for: > > > > bitbake virtual/kernel -c cleansstate; bitbake virtual/kernel > > > > These tests are based on the script linked from > > https://wiki.yoctoproject.org/wiki/Performance_Test where the kernel > > test is this is the second number above list, overall build time is the > > first. > > > > Have you any time to look into this performance regression? > > Well, patch replaced a call to os.path.realpath() with a more accurate > variant honoring a sysroot. There must be more work be done to replace > things previously solved by the operating system with custom (python) > code. > > I will try to write a C implementation of oe.path.realpath() but this > can take some time and I do not have an idea how to integrate it into > oe.
The issue is not an interpreted language verses C, the problem is the shear number of syscalls. Each isdir() and islink() call means a stat call into the kernel. For "bitbake virtual/kernel -c package", I'm seeing 400,000 stat calls for 23,000 files. We know the filesystem doesn't change so we could cache the stat calls and hopefully hence the speed. I actually have some code I tried this with but its not working as well as I'd like so I need to do some further investigation about what is happening. Cheers, Richard _______________________________________________ Openembedded-core mailing list [email protected] http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
