If you're using document() heavily, there are some known issues regarding whether/when/how-long a DTM is retained in memory. There's a PI that can be used as a workaround in the common case of an <xsl:for-each> with a document()-based expression in its select, but a general and automated answer is still being developed. (Basically, the pre-DTM default was not to cache source documents unless explicitly directed otherwise; the DTM-based code is currently defaulting the other way.)
Outside of that, I don't think we're aware of any particular memory problems; indeed, some users have said the DTM model appears to be saving them a significant amount of space. If you've got an isolatable testcase (perferably one that can be run from the command line rather than needing a complex surrounding environment), we could run it through a memory usage analyser and see where the space is actually going.
