Noel J. Bergman wrote:
I though that I should have tried to drop the "wrapping/delegation"
behaviour in favor of a simple extension.
I recall thinking the same. And moving to a shared inputstream and
streaming should be a big win on footprint. Dan also mentioned it, with
respect to Derby.
I just reread a thread (10-25 January 2006) with comments and code by
Markus Kuhn and Ofir Gross and maybe we should also incorporate some of
their suggestions. E.g: we could provide more constructors to
MimeMessageWrapper so that we can use it instead of MimeMessage when
"cloning" other MimeMessages.
Then we could simply fallback to a temporary sharedfile for the
contentStream as like as we do while receiving smtp messages with
MimeMessageInputStreamSource (that I already converted to provide
sharedinputstreams, in my test).
This way it was simpler to define when the message is needed and I've
been able to avoid the message loading for most header operations
(previously needed).
This is a large part of what the wrapper was originally written to do, IIRC.
Way before I started with JAMES.
Yes, but previously the full message (instead of headers) was loaded for
each setHeader operation. Now also set operations on headers try to
avoid the message loading.
I made [streaming] optional by adding a new "getBody" option
(useBinaryStream) to the dbOptions in sqlResources so that
testers could try this feature over different dbs and provide
feedback.
If you can complete the changes, I'd be in favor of getting them into the
next milestone.
I've done more tests with db repositories (not dbfile) over mysql, derby
and oracle and big messages (100MB+) and limited Xmx (40MB for mysql
and derby, 50MB for oracle and hsqldb) and I found something:
I need to use a PipedInputStream/PipedOutputstreams with an additional
buffering thread in order to store the message using streams for both
mysql and derby. I tested another option (setBinaryStream on a blob
obtained with "select for update") where this was not needed but both db
throws various "not implemented" exceptions.
IMHO this is not a showstopper because Activation framework (JAF)
already spawn similar threads when working with mimemessages. The main
problem is that the exception handling is much more difficult with this
kind of "worker" threads around. On the performance side we probably
want to run the thread when the message is larger of, as an example,
10KB. With smaller messages we can use a bytearray buffer and avoid the
additional thread (I prefer to add this easy optimization after we
tested to correctness of the streaming methods).
On the mysql side the streaming operations works correctly only using
mysql 4.1+, connector/j 5+ (latest connector/j 3.1.12 still have bugs)
and adding an "emulateLocators=true" parameter in the jdbc url.
The bad news is that using the streaming methods with older
connectors/servers ends in corrupted/truncated data!
Testing hsqldb I get OutOfMemoryExceptions and Corrupted Data all the
times so I skipped it.
The great news is that the same identical queries works with Oracle too!
I still can't fully believe this thing (I work often with jdbc/oracle
and I always had to write specific code). I only tested oracle 9.2.0.4
accessed by the latest 10G thin jdbc driver (10.1.0.4.0) (older drivers
didn't work).
I changed the MimeMessageJDBCSource to provide a SharedInputStream from
the db, but I'm not sure this is a good idea: currently if a message
created from that source is parsed javamail open a log of new shared
streams (creates new connection/queries to db). Maybe I should try to
intercept the parsing and copy the stream to a temporary shared file
instead of opening new connections (not totally sure it would be better)
Now I need a few (many?!) hours to put order on what I wrote and I hope
I'll commit something soon. After the commit and a few more tests we'll
vote on what to do with defaults. We probably should defaults to
binarystreams for derby, as we exactly know what version of derby is
used, and use standard bytearray operations for oracle and mysql and
perhaps add a note in the config.xml explaining how to enable stream
operations and what driver version/server version is needed.
As a side note I've seen that other players (activemq and jboss mail
server) are using another approach to save big messages: they split the
message over multiple records. I personally don't like this solution and
I prefer a big restriction on the usable jdbc drivers but a full
streaming support or as a last resort the file (or dbfile) repository.
Stefano
PS: I would like to alter the SMTPServer to directly stream incoming
messages to DB, but unfortunately I found out that we have to know the
exact size of the message before being able to stream it to db!
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
