Hello, I followed this IoBuffer discussion for some time now and I have a general concern:
I don't see a need to deal with buffers at all outside the framework or even invent a buffer api a developer has to learn. What about implementing the whole buffer thing as a non blocking composite stream? The first thing is: You said, that there is a need to not expose the API too much to prevent mistakes. When using an IoInputStream version of a composite buffer for example, there is no need to deal with position, limit, mark etc. Everything is just read once and mina could convert input to/from output streams for interal population or for giving them to protocols in the right type. The next thing is the composite buffer idea: When using streams, complete or partial basic byte arrays can be appended with zero-copy. The only thing that needs to be done is take care of this when reading data from the combined byte arrays to merge them if necessary e.g. for readInt(), readString() etc.. You could even slice the underlying byte array arrays into another stream with appropriate methods. Buffers could even overlap between the original and the sliced buffer, assuming that no more data is written to the sliced one (it's a stream, pure IoInputStream without put-methods). Another advantage is, that the coherent composite stream could be reused after a decoder read the available bytes and filled with more data as it comes in so that there would no need for a developer to handle buffering on its own. He just checks if enough data is available (e.g. for the header of a binary protocol or a string for a text based one) and reads it - the decoder just needs to be notified when new data is available. If only partially enough content is available, the not yet consumed bytes are kept inside the stream which is filled more the next time data is read. Also completely consumed byte arrays could be freed automatically by removing the reference from the stream. On this base, mina could populate an IoOutputStream for each connection and fill it with byte arrays without a need to copy anything. If just appends basic byte arrays to it. The decoder is notifed when new data is available so it can check if it is able to continue to process (e.g. the stream emmits an EOFException if not enough data is available on readString() or a binary decoder just checks for the available() byte count).For this Mina converts the IoOuputStream to an IoInputStream (new IoInputStream(output), zero copy of course, same byte array arrays) and gives it to the decoder for further processing which generates some sort of messages for decoderoutput - you all know the procedure. When this is done, the stream is reused (new IoOutputStream(input)) and filled even more with byte arrays. All not yet consumed data is kept available for the decoder. If a developer creates a new IoOutputStream, large enough byte arrays could just be appended. For small values (e.g. for putInt() etc.), the stream could expand by just adding another buffer of a predefined size (e.g. 128 bytes on the default or explicitly set by the developer if he is concerned of performance) for a sequence of put operations - so there is still an easy possibility for the extension mechanism. So: Almost every java guy knows how to deal with streams and the composite and api issues would be solved, too. It should not be too hard to add all the put/get Byte/Short/Int/Long/String methods to it to read data types from overlapping byte arrays. The available() method to count the remaining bytes also can be cached on read/write so that no costly iteration over byte array arrays (I start to like this phrase) needs to be done. However, I am very new to mina and I may just miss the point but for my binary FastCGI implementation this kind of streams would satisfy all my needs. For example: First a 8 bytes long header is read containing some protocol specific stuff and the content length. Afterwards, when enough content bytes are available, the stream is added to the message as the content (slice it to fixed length) and the message containing the sliced stream is written to decoderoutput. Afterwards, decoding starts again on the still existing composite stream and nothing had to be copied. The buffers at the end of the stream (that may contain the last content bytes and a new header) could even overlap between duplicates/slices without a possibility to modify the wrong bytes because of the nature of a stream, as mentioned above. Afaik this works the same way for HTTP (mixed text/binary) and text based protocols would be even simpler as they already are because they could just check if a string is available (this also could be cached internally to not check for (CR)LF on already checked bytes when the appropriate getString() method is called another time). As I already said I don't see a need to struggle with buffers at all on user/protocol developer side, so wouldn't this be the best way? reagards Daniel
