Hey Rich,
Oh sure. I know you spent great deal of time to resolve this issue. :)
It would be great if you can attach the patch again to our JIRA page just
to make sure it's correctly licensed.
Thanks again!
On Thu, 15 May 2008 21:19:26 +0900, Rich Dougherty <[EMAIL PROTECTED]> wrote:
Hi Trustin
Not just discussion! Since my last email I've spent a bit of time playing
around with a set of classes and interfaces that (hopefully) provide the
sort of functionality that we talk about. They're not quite ready, but I
think it might be worth sharing them, so you can consider them for your
new
branch.
I've attached the source for these files. They're not complete yet or
very
well tested (e.g. flushing doesn't work properly), but I think I'm
reasonably happy with the design. Notably there's no mark/reset
functionality, nor is there support for reading/writing a wide variety of
types (just byte/ByteBuffer/int). These features should be easy to add. I
just wanted to keep it as simple as possible initially.
The key thing I've done is factored the design into multiple classes and
interfaces. This move away from a single, monolithic class is
intentional.
Having multiple classes allows users to pick the implementation that
suits a
particular usage. It also dramatically simplifies implementation since we
can use lower-level classes to build our higher-level ones, rather than
trying to do everything within a single class. e.g. Stream classes are
much
easier to write once a good composite buffer class has been written.
Here's a summary of the classes: (Names are just placeholders.)
- *BufferByteArray:* A class that wraps ByteBuffer, providing simple
utility
methods and especially a *free* method to support pooling.
- *CompositeByteArray:* There is a class with the same interface that
supports multiple buffers with O(1) adding and removing.
- **ByteArray.Cursor: *Stores position information for a *ByteArray*.
Keeping this information separate makes the classes simpler, and gives
users
more flexibility (e.g. reading and writing at separate positions at the
same
time).
- *CompositeByteArrayRelativeReader/Writer: *Restrictive,
relative-access-only stream interfaces, backed by a *CompositeByteArray*.
The benefit of these stream interfaces is that they control access to the
underlying buffers, and so can do certain things automatically for the
user
(e.g. freeing buffers).
Anyway the design should allow all the big features we've been talking
about:
- zero-copy reads
- gathering writes
- optional asynchronous stream interface with
auto-freeing/auto-allocating/auto-flushing of ByteBuffers
Please forgive me for not providing any more detail in this email - it's
past midnight here, so I've had to be brief. :-) I've at least gone
through
the classes and given them some light javadoc, so hopefully that will be
good enough for the moment.
Cheers
Rich
On Thu, May 15, 2008 at 8:35 PM, 이희승 (Trustin Lee) <[EMAIL PROTECTED]>
wrote:
If there's no objection or no better alternative, let me create a branch
and go forward. The implementation should be straightforward. I will
also
give you updates whenever meaningful commit is made.
A bunch of thanks to Daniel, Rich and Emmanuel for all the related
discussion! :)
Cheers,
On Wed, 14 May 2008 23:31:00 +0900, 이희승 (Trustin Lee)
<[EMAIL PROTECTED]>
wrote:
Daniel and folks,
It seems like we are getting close to consensus.
I concur with you that the stream-like implementation makes a perfect
sense. I'd prefer to roll out our own stream-like real non-blocking
stream
and provide a wrapper to InputStream and OutputStream. I think we can
use
java.io.RandomAccessFile as a good starting point:
* http://java.sun.com/javase/6/docs/api/java/io/RandomAccessFile.html
What we need with the new interface are:
* stream-like sequential read and write operations for primitive
types,
arrays of primitive type and ByteBuffer
* seek operations - mark, reset, position (let's keep this minimal
because it's not used often)
* discard operation for efficient memory utilization (+ discardAll)
* no flip / no compact
* no convenience read / write methods such as readString, readUTF8,
readEnum, ... (will be provided in a utility class)
For instance, it could look like this:
public interface NameToBeDetermined extends Iterable<ByteBuffer> {
ByteBuffer read();
byte readByte();
int readInt();
int read(ByteBuffer dst);
...
void write(ByteBuffer src);
void writeByte(byte value);
void writeInt(int value);
...
int available();
int position();
void position(int position);
int skip(int bytes);
void mark();
void reset();
void discard();
void discardAll();
// Some extra methods which might help:
ByteOrder order();
void order(ByteOrder order);
boolean isAutoDiscard();
// true by default, which is OK for most users.
void setAutoDiscard(boolean autoDiscard);
// Conversion methods...
ByteBuffer[] asByteBuffers();
InputStream asInputStream();
OutputStream asOutputStream();
}
I introduced autoDiscard so any read buffer is discarded automatically.
Some users would want this happen manually, then they can call
setAutoDiscard(false).
Another good thing to have is a bounded stream, which has limited
capacity
to prevent OutOfMemoryError. It should be simple to implement because
all
it needs to is implement the interface.
The last (?) thing to consider is to use get and put instead of read
and
write to ease migration from IoBuffer, and it is easier to type.
WDYT?
On Sat, 03 May 2008 23:49:41 +0900, Daniel Wirtz <[EMAIL PROTECTED]>
wrote:
Hello Trusin,
you pointed out some very interesting thoughts. In fact it will
result in
unexpected behaviour, when common stream wrappers are used on a
non-blocking
stream without proper orevious checking. I see two possibilities:
Using
streams that emulate a blocking mode and throw some sort of
WouldBlockException if blocking is disabled (by default) or rolling
out
our
own stream-like version of a real non-blocking stream.
I see the following general advantages of streams over buffers:
- Very easy to use because the stream API is very clear and known by
most
developers
- Automatic dispose of already consumed underlying buffers
- No exposure of unnecessary low level functionality like flip(),
limit(), iterate over buffers etc.
- Possibility to reuse the same stream when some new data is
available so
that protocol developers don`t need to handle buffering on their own
(just
leave unused bytes inside the stream and process them later when more
data
is available / the decoder is notified)
- Zero-Copy without a need for users to know (much) about the
underlying
buffer architecture (just append buffers to a stream)
- Not an advantage, but also: Possibility for random access by using
mark(), skip() and reset() in sequence, maybe with an additional
moveTo()
relative to the marked position implemented by combining the previous
three
- however, random access is not required for most protocols
- Ability to use common wrappers for OutputStream
- Ability to use common wrappers for InputStream (but only if data
availability is previously checkable/checked to be valid)
Disadvantages of extending java.io.InputStream:
- Unexpected behaviour when using common wrappers on a stream in
non-blocking mode with no proper previous checking of available data
So, do we really need to extend InputStream / use common input
wrappers?
Maybe it makes sence to roll out our own stream-like real non-blocking
inputstream that cannot be used with common input wrappers but
implementing
all the required put- and get-Methods. However, I still think that a
stream-like implementation of the whole buffer thing makes it much
easier
for the end user to understand and use.
regards
Daniel
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/
--
http://www.richdougherty.com/
--
Trustin Lee - Principal Software Engineer, JBoss, Red Hat
--
what we call human nature is actually human habit
--
http://gleamynode.net/
