inserting handler/filter in main server vs. virtual host configuration?

[Jeff Ambrosino]

For httpd.conf in Apache 2.x / mp2, does it matter if I insert a
handler in the main server section vs. a virtual host section?  My
question pertains specifically to performance/memory, as I already
understand the logical implications.

PerlChildInitHandler +MyStuff::InitHandler

vs.

<VirtualHost>
    PerlChildInitHandler +MyStuff::InitHandler
</VirtualHost>

thanks
JB