On Thu, 2010-08-26 at 13:49 +0100, Richard Purdie wrote: > The bad news is that parsing took a factor of 10 times longer :(. > Profiling reveals its the python function parsing that is taking the > majority of the time. As an idea of numbers: > > Parsing 85 recipes went from taking 4 seconds to 40 seconds. That > parsing involved parsing 8500 python functions and in doing so, the main > time was spent looping over 1.4 million tokens that those python > functions split into. > > By swapping NodeVisitor for ast.walk, I was able to get 10% speed back > but the main drain is the 1.4 million tokens which we have to iterate > over. I think it should be possible to put a cache around the python > parsing code and reduce the number of real parsing calls so I will look > into that next.
To update, adding a cache around the python parser resulted in 1.4 million interations dropping to 272000 and it was only 4.5 times slower. Time wise, under profiling this lead to python_parse and shell_parse both taking 17s. In python_parse 5s was in spent in dedent_python() which looked like an obvious place to improve. I've written a different dedent_python() implementation which takes 0.12s for the same workload so python_parse is now 12s. This brings us to being 4 times slower. So still not good but better. Cheers, Richard _______________________________________________ Bitbake-dev mailing list [email protected] https://lists.berlios.de/mailman/listinfo/bitbake-dev
