Hi (quickly reading ,
reading everything-you-can-get might starve the application logic.
We currently have some "realtime" stuff which must be transferred as quickly as
possible, but it's just some bytes (Biggest messages are 1K, smallest about 10
bytes). This logic would increase roundtrip times to numbers where we can shut
our servers down.
In such a setup it would be nice if every 1K ByteBuffer is pushed to the chain,
since in most cases it's a full message and waiting any longer just increases
roundtrip times.
In this case, streaming big data would be very inefficient, so don't expect a
simple solution that fits all problems.
Maybe the application/decoder logic should set some hints to the Processor on a
session base. This way you could even switch a running session between short
reaction time and efficient streaming.
A quick and unfinished thought about a hint-class:
class DecodingHints {
static DecodingHints MASS_DATA = new DecodingHints(65535, 10)
static DecodingHints NORMAL = new DecodingHints(16384, 10)
static DecodingHints QUICK = new DecodingHints(1024, 1)
DecodingHints(int bufferSize, in maxBufferedBuffersCount){
...
}
}
Usage:
class MyDecoder {
...
if (isStreamingBegin){
session.setDecodingHints(DecodingHints.MASS_DATA);
} else if (isStreamingEnd) {
session.setDecodingHints(NORMAL);
}
...
}
> Chad Beaulac [mailto:[email protected]] wrote
>
> Hi Emmanuel,
>
> A 1k ByteBuffer will be too small for large data pipes. Consider using
> 64k
> like you mentioned yesterday.
> Draining the channel before returning control to the program can be
> problematic. This thread can monopolize the CPU and other necessary
> processing could get neglected. The selector will fire again when
> there's
> more data to read. Suggest removing the loop below and using a 64k
> input
> buffer.
>
> Regards,
> Chad
>
>
> On Thu, Dec 1, 2011 at 4:00 AM, Emmanuel Lecharny
> <[email protected]>wrote:
>
[snip]
> >
> > As you can see, instead of reading one buffer, and call the chain, we
> > gather as many data as we can (ie as many as the channel can
> provide), and
> > we call the chain.
> > This has one major advantage : we don't call the chain many times if
> the
> > data is bigger than the buffer size (currently set to 1024 bytes),
> and as a
> > side effect does not require that we define a bigger buffer (not
> really a
> > big deal, we can afford to use a 64kb buffer here, as there is only
> one
> > buffer per selector)
> > The drawback is that we allocate ByteBuffers on the fly. This can be
> > improved by using a pre-allocated buffer (say a 64kb buffer), and if
> we
> > still have something to read, then we allocate some more (this is
> probably
> > what I will change).
> >
> > The rest of the code is not changed widely, except the decoder and
> every
> > filter that expected to receive a ByteBuffer (like the
> LoggingFilter). It's
> > just a matter of casting the Object to IoBuffer, and process the
> data, as
> > the IoBuffer methods are the same than the ByteBuffer (except that
> you
> > can't inject anything but ByteBuffers into an IoBuffer, so no put
> method,
> > for instance).
> >
> > The decoders for Http and Ldap have been changed to deal with the
> > IoBuffer. The big gain here, in the Http cas, is that we don't have
> to
> > accumulate the data into a new ByteBuffer : the IoBuffer already
> accumulate
> > data itself.
> >
> > The IoBuffer is stored into the session, which means we can reuse it
> over
> > and over, no need to create a new one. I still have to implement the
> > compact() method which will remove the used ByteBuffers, in order for
> this
> > IoBuffer not to grow our of bounds.
> >
> > thoughts, comments ?
> >
> > Thanks !
> >
> > --
> > Regards,
> > Cordialement,
> > Emmanuel Lécharny
> > www.iktek.com
> >
> >
>
--------------------------------------------------------------------------
PROEMION GmbH
Steve Ulrich
IT Development (IT/DEV)
Donaustrasse 14
D-36043 Fulda, Germany
Phone +49 (0) 661 9490-601
Fax +49 (0) 661 9490-333
http://www.proemion.com
Geschäftsführer: Dipl. Ing. Robert Michaelides
Amtsgericht-Registergericht-Fulda: 5 HRB 1867
--------------------------------------------------------------------------
E-mail and any attachments may be confidential. If you have received this
E-mail and you are not a named addressee, please inform the sender immediately
by E-mail and then delete this E-mail from your system. If you are not a named
addressee, you may not use, disclose, distribute, copy or print this E-mail.
Addressees should scan this E-mail and any attachments for viruses. No
representation or warranty is made as to the absence of viruses in this E-mail
or any of its attachments.
AKTUELLES:
http://www.proemion.de
NEWS:
http://www.proemion.com
