OutOfMemory means you've run out of Java heap space. Most JVMs have parameters that will let you increase the maximum size of the heap; you can start by raising that limit and seeing if it gets you over the hump. I normally run with -mx64M -Xmx64M (only one of these is actually needed, but which one depends on which JVM you're using).
Search past discussion in the xalan-j-dev list for discussion of "pruning" for comments about the more general issue of how to process huge documents efficiently. We have some ideas in this area -- and I've got an early draft of some code which ought to help take us in that direction -- but nothing's currently checked in other than a highly specific workaround for the case of for-each with select based on document().
