Daniel Kulp wrote:
I did quite a bit of work yesterday to fix some areas where large
messages were causing problems. They all pretty much centered around
mis-use of the CachedOutputStream (assuming it's always a
ByteArrayOutputStream which isn't true once it flips to a file).
However, while digging through those, I've noticed a few other potential
problems that I'd like to discuss.
1) Logging interceptors - right now, we log the entire message to the
Logger. However, the logger requires a String, so the entire message
is converted to a String (keep in mind, a String is utf-16 so twice the
memory is consumed) for logging. Plus, do you really think logging a
10MB message to the Logger is the best option? I'm going to add a
configurable "limit" that will Log just the first "limit" bytes of the
message. Should the default be -1 (no limit) or something reasonable
like 4K? Also, we could have a configuration to allow it to
optionally log to a separate file instead of the Logger. We can then
directly copy from stream to stream with very little memory usage.
That's going to be a lot trickier though as we'll have to deal with
concurrent access to the file from multiple threads and such.
Maybe a more reasonable limit would be 100 K and a warning message
otherwise.
2) JMS transport - the JMS transport always deals with the entire message
as either a String (for TextMessage) or byte[]. That seems to be a
limitation for standard JMS API's. Some vendors (such as ActiveMQ) do
have proprietary extensions to allow some proprietary stuff to help.
Examples:
http://activemq.apache.org/blob-messages.html
http://activemq.apache.org/jms-streams.html
It might be good to allow some extra hooks into the JMS transport to
allow for easy subclassing to enable some of the hooks. (That said,
I'd like to see the JMS transport also changed over to using the Spring
JMS configuration.)
+1 on both accounts
3) WS-RM - The RMMessage object holds the message as a byte[]. Thus,
it's entirely in memory until it's stored. It would probably be a good
idea for this to be passed as the original CachedOutputStream and stored
directly to/from that stream
Yeah, I suppose we could stream it to its storage location and then
stream it back out later on. Never thought about that, but that makes
perfect sense now that you mention it!
- Dan
--
Dan Diephouse
MuleSource
http://mulesource.com | http://netzooid.com/blog
