On Thu, Aug 26, 2010 at 7:56 AM, Richard Purdie <[email protected]> wrote:
> On Thu, 2010-08-26 at 13:49 +0100, Richard Purdie wrote: > > The bad news is that parsing took a factor of 10 times longer :(. > > Profiling reveals its the python function parsing that is taking the > > majority of the time. As an idea of numbers: > > > > Parsing 85 recipes went from taking 4 seconds to 40 seconds. That > > parsing involved parsing 8500 python functions and in doing so, the main > > time was spent looping over 1.4 million tokens that those python > > functions split into. > > > > By swapping NodeVisitor for ast.walk, I was able to get 10% speed back > > but the main drain is the 1.4 million tokens which we have to iterate > > over. I think it should be possible to put a cache around the python > > parsing code and reduce the number of real parsing calls so I will look > > into that next. > > To update, adding a cache around the python parser resulted in 1.4 > million interations dropping to 272000 and it was only 4.5 times slower. > > Time wise, under profiling this lead to python_parse and shell_parse > both taking 17s. In python_parse 5s was in spent in dedent_python() > which looked like an obvious place to improve. > > I've written a different dedent_python() implementation which takes > 0.12s for the same workload so python_parse is now 12s. > > This brings us to being 4 times slower. So still not good but better. Interesting, is your version of dedent_python available somewhere? -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
_______________________________________________ Bitbake-dev mailing list [email protected] https://lists.berlios.de/mailman/listinfo/bitbake-dev
