It depends alot on how you are loading those modules too. I try to use a load on request for the less used classes (using autoload) and load the heavily used classes in the bootstrap so they are available to the entire app.
Major speed improvements when I use static methods in my own classes instead of instantiating objects... Zend_Log has been notoriously heavy for me; even though it is really powerful I just ended up rolling my own very simple and fast logging module. You might check out Picora front controller framework to replace Zend_Controller_Front (since that is heavy too), Picora is much much smaller and quicker - but less features. I also use my own home brew view system, although it uses XSLT transforms to generate output, so it tends to be slower than the stock solution with really big/complex pages.
