David M. Lloyd wrote:
On 04/28/2008 10:46 AM, Emmanuel Lecharny wrote:
David M. Lloyd wrote:
- should we serialize the stream at some point ?
What do you mean by "serialize"?
Write to disk if the received data are too big. See my previous point
(it's up to the decoder to deal with this)
Ah I understand. Yes, it would be up to the decoder. Though
hopefully most decoders can process the entire buffer without having
to keep the buffer data around. The faster you can get rid of
incoming buffers, the less memory you will consume overall.
+1
Note that a buffer might contain data from more than one message as
well. So it's important to use only a slice of the buffer in this case.
Not a big deal. Again, it's the decoder task to handle such a case.
We have experimented such a case in LDAP too.
Yes and no - we should at least make support methods available to make
this easier. MINA won't win any fans by being labeled "framework that
makes decoder implementers do the most work" :-)
The best would be to offer an API having such a method : a blocking
getNextByte() method. Atm, you have to do something like
BB.hasRemaining() before grabbing a byte. Painfull ... and ineficient !
- how to write an efficient encoder when you have no idea about the
size of the data you are going to send ?
Use a buffer factory, such as IoBufferAllocator, or use an even
simpler interface like this:
public interface BufferFactory {
ByteBuffer createBuffer();
}
[..]
That's an idea. But this does not solve one little pb : if the reader
is slow, you may saturate the server memory with prepared BB. So you
may need a kind of throttle mechanism, or a blocking queue, to manage
this issue : a new BB should not be created unless the previous one
has been completely sent.
Well that would really depend on the use case. If you're sending the
buffers as soon as they're filled, it might not be a problem.
I don't think so. If you do that, you may stack thousands of BB if the
client reader is slow. Likely to be a real problem quite fast ... Unless
if you meant "sending to the client".
If you saturate the output channel though, you'll have to be able to
handle that situation somehow. It ultimately has to be up to the
protocol decoder to detect a saturated output channel and perform
whatever action is necessary to squelch the output message source
until the output channel can be cleared again.
I would prefer the underlying layer to handle this case. The encoder is
responsible to encode the data, not to handle the client sloppiness. So
when you have filled a BB to be sent to the client, just push it to the
underlying layer, and wait for this layer to tell you that you can
provide one more BB, and so on. With a correct time out, you may even
kill the encoder task, to free some ressource for serious clients :)
The right answer might be to spool output data to a disk-backed queue,
or it might be to block further requests, etc. Basically the same
situation that exists today.
At some point, we need to establish some policy about how to handle such
problems. Even a disk is a limited ressource (yeah, you can still create
gmail account, and store data as mails, as each gmail account allows you
to store 2 Gb or more, but sooner or later they will knock at your door ;)
(this idea has been patented by me : "An almost unlimited storage using
GMail accounts")
Another option is to skip ByteBuffers and go with raw byte[] objects
(though this closes the door completely to direct buffers).
Well, ByteBuffers are so intimately wired with NIO that I don't think
we can easily use byte[] without losing performances... (not sure
though ...)
Not that I'm in favor of using byte[] directly, but it's easy enough
to wrap a (single) byte[] with a ByteBuffer using the
ByteBuffer.wrap() methods.
Yep.
Yet another option is to have a simplified abstraction for byte
arrays like Trustin proposes, and use the stream cleasses for the
buffer state implementation.
This is all in addition to Trustin's idea of providing a byte array
abstraction and a buffer state abstraction class.
I'm afraid that offering a byte[] abstraction might lead to more
complexity, wrt with what you wrote about the way codec should handle
data. At some point, your ideas are just the good ones, IMHO : use
BB, and let the codec deal with it. No need to add more complex data
structure on top of it.
Maybe. If this route is taken though, a very comprehensive set of
"helper" methods will probably be needed. ByteBuffer has a pretty
lousy API. :-)
ByteBuffers are, hmmm, buffers :)
So I'd cast my (useless and non-binding) vote behind either using
ByteBuffer with static support methods, or using a byte array
abstraction object with a separate buffer abstraction like Trustin
suggests.
Some experiment with code could help, at this point. Navigating at such
high level does not help a lot to grasp the real bytes we are dealing
with on the network level ;)
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
