Re: [MINA 3.0] IoBuffer usage

2012-01-17 Thread Chad Beaulac
On Mon, Jan 16, 2012 at 1:10 PM, Emmanuel Lecharny elecha...@gmail.comwrote: On 1/16/12 2:56 PM, Chad Beaulac wrote: Emmanuel, (all) I'm working on this Camel ticket: https://issues.apache.org/**jira/browse/CAMEL-2624https://issues.apache.org/jira/browse/CAMEL-2624 I finished the initial

Re: [MINA 3.0] IoBuffer usage

2012-01-17 Thread Emmanuel Lécharny
On 1/17/12 3:02 PM, Chad Beaulac wrote: On Mon, Jan 16, 2012 at 1:10 PM, Emmanuel Lecharnyelecha...@gmail.comwrote: On 1/16/12 2:56 PM, Chad Beaulac wrote: Emmanuel, (all) I'm working on this Camel ticket:

Re: [MINA 3.0] IoBuffer usage

2012-01-17 Thread Chad Beaulac
On Jan 17, 2012, at 9:32 AM, Emmanuel Lécharny elecha...@apache.org wrote: On 1/17/12 3:02 PM, Chad Beaulac wrote: On Mon, Jan 16, 2012 at 1:10 PM, Emmanuel Lecharnyelecha...@gmail.comwrote: On 1/16/12 2:56 PM, Chad Beaulac wrote: Emmanuel, (all) I'm working on this Camel ticket:

Re: [MINA 3.0] IoBuffer usage

2012-01-17 Thread Emmanuel Lecharny
IFAICT, on MINA 2, you should not have any issue. The loop where we write buffers into the channel is : ... final int maxWrittenBytes = session.getConfig().getMaxReadBufferSize() + (session.getConfig().getMaxReadBufferSize() 1); int writtenBytes = 0;

Re: [MINA 3.0] IoBuffer usage

2012-01-17 Thread Chad Beaulac
Ok. I'll test the throughput on it with the tests I have. Thanks Chad Sent from my iPad On Jan 17, 2012, at 11:33 AM, Emmanuel Lecharny elecha...@gmail.com wrote: IFAICT, on MINA 2, you should not have any issue. The loop where we write buffers into the channel is : ...

Re: [MINA 3.0] IoBuffer usage

2012-01-17 Thread Emmanuel Lecharny
On 1/18/12 2:57 AM, Chad Beaulac wrote: Ok. I'll test the throughput on it with the tests I have. Let me know if you have any isues with the throughput you obtain. Be sure to set the default read buffer size to the maximum size, AFAIR, it's set to 2048 bytes in MINA 2 (and can't be higher

Re: [MINA 3.0] IoBuffer usage

2012-01-16 Thread Chad Beaulac
Emmanuel, (all) I'm working on this Camel ticket: https://issues.apache.org/jira/browse/CAMEL-2624 I finished the initial cut of https://issues.apache.org/jira/browse/CAMEL-3471 to create a mina2 component in Camel. CAMEL-2624 adds the full async behavior for Mina2 endpoints in Camel. Would it

Re: [MINA 3.0] IoBuffer usage

2012-01-16 Thread Emmanuel Lecharny
On 1/16/12 2:56 PM, Chad Beaulac wrote: Emmanuel, (all) I'm working on this Camel ticket: https://issues.apache.org/jira/browse/CAMEL-2624 I finished the initial cut of https://issues.apache.org/jira/browse/CAMEL-3471 to create a mina2 component in Camel. CAMEL-2624 adds the full async

Re: Re: [MINA 3.0] IoBuffer usage

2011-12-05 Thread Julien Vermillard
Hi, snipped a lot of the message :) So, do you mean that the underlying layer will not allow us to push say, 20M, without informing the session that it's full ? In other word, there is a limited size that can be pushed and we don't have to take care of this limit ourselves ? Sort of. If

Re: [MINA 3.0] IoBuffer usage

2011-12-05 Thread Emmanuel Lécharny
On 12/4/11 2:34 PM, Chad Beaulac wrote: Hi Chad, Agreed :-) Do you have a Git repo setup for Mina3? I'll help you write some of it if you like. With unit tests, of course. ;-) http://git.apache.org/ This is a read only Git repos. Internally, we are -still- using SVN (but we are

Re: Re: [MINA 3.0] IoBuffer usage

2011-12-05 Thread Chad Beaulac
Looks perfect except one thing. Don't allow clients to put WriteRequest's into the queue while you're draining it. If you allow other threads to enqueue more WriteRequest's, the algorithm is unbounded and you risk getting stuck writing for only one channel. I added a synchronized block below. On

Re: [MINA 3.0] IoBuffer usage

2011-12-05 Thread Emmanuel Lecharny
On 12/5/11 2:39 PM, Chad Beaulac wrote: Looks perfect except one thing. Don't allow clients to put WriteRequest's into the queue while you're draining it. If you allow other threads to enqueue more WriteRequest's, the algorithm is unbounded and you risk getting stuck writing for only one

Re: [MINA 3.0] IoBuffer usage

2011-12-05 Thread Chad Beaulac
Excellent. I like the counter in the write algorithm. Sent from my iPhone On Dec 5, 2011, at 8:50 AM, Emmanuel Lecharny elecha...@gmail.com wrote: On 12/5/11 2:39 PM, Chad Beaulac wrote: Looks perfect except one thing. Don't allow clients to put WriteRequest's into the queue while you're

Re: [MINA 3.0] IoBuffer usage

2011-12-05 Thread Julien Vermillard
since it's a ConcurrentLinkedQueue it could be a perf killer to do a .size() from the oracle javadoc : Beware that, unlike in most collections, the size method is NOT a constant-time operation. Because of the asynchronous nature of these queues, determining the current number of elements requires

Re: [MINA 3.0] IoBuffer usage

2011-12-05 Thread Emmanuel Lecharny
On 12/5/11 3:50 PM, Julien Vermillard wrote: since it's a ConcurrentLinkedQueue it could be a perf killer to do a .size() from the oracle javadoc : Beware that, unlike in most collections, the size method is NOT a constant-time operation. Because of the asynchronous nature of these queues,

Fwd: Re: [MINA 3.0] IoBuffer usage

2011-12-04 Thread Emmanuel Lecharny
Posted on the wrong mailing list... Forwarding there. Hi Chad, On 12/4/11 1:25 AM, Chad Beaulac wrote: A single algorithm can handle large data pipes and provide extremely low latency for variable, small and large message sizes at the same time. AFAIU, it' snot because you use a big

Re: Re: [MINA 3.0] IoBuffer usage

2011-12-04 Thread Chad Beaulac
On Sun, Dec 4, 2011 at 8:04 AM, Emmanuel Lecharny elecha...@gmail.comwrote: Posted on the wrong mailing list... Forwarding there. Hi Chad, On 12/4/11 1:25 AM, Chad Beaulac wrote: A single algorithm can handle large data pipes and provide extremely low latency for variable, small

Re: [MINA 3.0] IoBuffer usage

2011-12-02 Thread Chad Beaulac
I think I'm not communicating my thoughts well enough. A single algorithm can handle large data pipes and provide extremely low latency for variable, small and large message sizes at the same time. On the Producer side: Application code should determine the block sizes that are pushed onto the

Re: [MINA 3.0] IoBuffer usage

2011-12-02 Thread Emmanuel Lécharny
On 12/2/11 2:02 PM, Chad Beaulac wrote: I think I'm not communicating my thoughts well enough. Well, I hope I have undesrtood what you said, at least :) A single algorithm can handle large data pipes and provide extremely low latency for variable, small and large message sizes at the same

[MINA 3.0] IoBuffer usage

2011-12-01 Thread Emmanuel Lecharny
Hi guys, yesterday, I committed some changes that make the NioSelectorProcessor to use the IoBuffer class instead of a singe buffer to store the incoming data. Here is the snippet of changed code : int readCount = 0; IoBuffer

Re: [MINA 3.0] IoBuffer usage

2011-12-01 Thread Chad Beaulac
Hi Emmanuel, A 1k ByteBuffer will be too small for large data pipes. Consider using 64k like you mentioned yesterday. Draining the channel before returning control to the program can be problematic. This thread can monopolize the CPU and other necessary processing could get neglected. The

Fwd: Re: [MINA 3.0] IoBuffer usage

2011-12-01 Thread Emmanuel Lécharny
Forwarded to the ML Quick note below. Sent from my iPhone On Dec 1, 2011, at 7:51 AM, Emmanuel Lécharnyelecha...@apache.org wrote: On 12/1/11 1:32 PM, Chad Beaulac wrote: Hi Emmanuel, A 1k ByteBuffer will be too small for large data pipes. Consider using 64k like you mentioned

Re: Fwd: Re: [MINA 3.0] IoBuffer usage

2011-12-01 Thread Emmanuel Lecharny
On 12/1/11 2:49 PM, Emmanuel Lécharny wrote: Forwarded to the ML Quick note below. Sent from my iPhone On Dec 1, 2011, at 7:51 AM, Emmanuel Lécharnyelecha...@apache.org wrote: On 12/1/11 1:32 PM, Chad Beaulac wrote: Hi Emmanuel, A 1k ByteBuffer will be too small for large data

Re: [MINA 3.0] IoBuffer usage

2011-12-01 Thread Emmanuel Lecharny
On 12/1/11 5:28 PM, Steve Ulrich wrote: Hi (quickly reading , reading everything-you-can-get might starve the application logic. We currently have some realtime stuff which must be transferred as quickly as possible, but it's just some bytes (Biggest messages are 1K, smallest about 10 bytes).

Re: [MINA 3.0] IoBuffer usage

2011-12-01 Thread Chad Beaulac
The reverse is true for the producer. Let's assume the writer/producer has a list of ByteBuffer. When the selector fires that you can write for the channel then: - write until there's nothing left to write, unregister for the write event, return to event processing - write until the the channel