Thanks for the quick reply. I've seen this problem on Mina 2.0.2 and 2.0.3.
Here is the pertinent code on the client side:
private void initConnector() {
connector = new NioSocketConnector();
connector.addListener(new MinaServiceAdapter() {
@SuppressWarnings("hiding")
@Override
public void sessionCreated(IoSession session) throws
Exception {
SocketClientPublisher.this.session = session;
}
@SuppressWarnings({ "hiding", "unused" })
@Override
public void sessionDestroyed(IoSession session) throws
Exception {
SocketClientPublisher.this.session = null;
}
});
connector.setHandler(new IoHandlerAdapter()); // ignoring
received messages
initChainBuilderDefaults(connector.getFilterChain());
}
private void initChainBuilderDefaults(DefaultIoFilterChainBuilder
chain) {
chain.addLast("codec", new
ProtocolCodecFilter(getPrefixedStringCodecFactory()));
chain.addLast("mdc", new MdcInjectionFilter());
chain.addLast("logging", getLogFilter());
}
private PrefixedStringCodecFactory getPrefixedStringCodecFactory() {
PrefixedStringCodecFactory factory = new
PrefixedStringCodecFactory();
factory.setDecoderMaxDataLength(Integer.MAX_VALUE);
factory.setEncoderMaxDataLength(Integer.MAX_VALUE);
return factory;
}
And on the server side:
private void initSiriAcceptor() throws IOException {
siriAcceptor = new NioSocketAcceptor();
siriAcceptor.setHandler(new IoHandlerAdapter() {
@Override
public void messageReceived(IoSession session, Object
message) throws Exception {
//
}
});
initChainBuilder(siriAcceptor.getFilterChain());
openSiriAcceptor();
}
private void initChainBuilder(DefaultIoFilterChainBuilder chain) {
initChainBuilderDefaults(chain);
chain.addLast("codec", new
ProtocolCodecFilter(getPrefixedStringCodecFactory()));
}
private void initChainBuilderDefaults(DefaultIoFilterChainBuilder
chain) {
chain.addLast("mdc", new MdcInjectionFilter());
chain.addLast("logging", getLogFilter());
}
The purpose of this large string (xml) is to provide a baseline for Timetable
information in a standard format. There are nearly 1/2 million rows in the
database.
If you have any suggestions please let me know. I'm investigating ways of
breaking up the message prior to sending. Thanks again for the reply.
Burt Alexander
Senior Software Engineer
Level 4, 175 Eagle Street, Brisbane QLD 4000
Telephone: +61 7 3039 2553
Facsimile: +61 7 3229 3702
Website: www.sigtec.com
Please consider the environment before printing this email
CONFIDENTIALITY NOTE: The content of this email is confidential and may be
legally privileged. It is intended solely for the addressee(s). Access to this
email by anyone else is unauthorised. If you are not the intended recipient,
any disclosure, copying, distribution or any action taken or omitted to be
taken in reliance on it, is strictly prohibited and may be unlawful.
-----Original Message-----
From: Emmanuel Lecharny [mailto:[email protected]]
Sent: Tuesday, 31 May 2011 2:40 PM
To: [email protected]
Subject: Re: Data loss when sending large strings
On 5/31/11 12:05 AM, Burt Alexander wrote:
> Hi, I'm using MINA successfully with one exception. Sending large Strings
> between a NioSocketAcceptor& Connector (143 megs)
Do I have to understand that you are transmitting a 143 Mb String ???
If so, here is what is going to happen, if you use the
PrefixedStringCodecFactory : a buffer will be used to cumulate the
incoming bytes until everything has been received. This buffer will be
associated with the session, and resides in memory. That means you will
store a 143 Mb buffer in memory...
Add to that the conversion to a String when all the bytes have been
received, that means that at a point, you will suck 3 times 143 Mb of
memory (143 Mb for the buffer, and as a char is 2 bytes long, 2 times
143 Mb for the String).
Your JVM will be under extreme load for a moment.
> appears to have the effect of losing much of the message. The acceptor
> then waits for the 'rest' of the message, no exception is thrown.
> I'm using the PrefixedStringCodecFactory with the encoder and decoder max
> length set to Integer.MAX_VALUE. I'd like to be able to throttle the sending
> of the message but I'm unable to determine how I can go about doing it - I've
> investigated using an ExecutorFilter with an IoEventQueueThrottle, a
> WriteRequestFilter with the same, changing read& write buffer sizes all to
> no avail. Using tcp to monitor the connection I see output like the
> following when I lose data:
>
> tcp 0 453224 ::ffff:172.16.9.65:50561 ::ffff:172.16.9.59:33000
> ESTABLISHED 1149/java
>
> Any help will be most appreciated. Thanks.
Here, if you can provide the snippet of code you are using, that could
help. Please also provide the MINA version you are using.
All in all, I'd like to understand why you have to convert the big chunk
of data to a String. You may need to implement some other logic to
smoothly handled such a transfer. Anyway, I need more info to be able to
propose alternatives here.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
