Re: Application works with single IoSession, throws exception on client side with multiple sessions

[Laurent Cohen]

Hello Emmanuel,

Thanks for your reply. I'm perfectly comfortable with English, so I'd 
rather stick to it, in case anyone else could benefit from reading this 
thread.
As far as I know, the points you mentioned previously are already 
handled. The only thing I do not need to handle is the case where we 
have multiple messages per incoming buffer, because the application is 
built in such a way that a client always sends a single message, and 
always in response to a server command.
The code I'm using is essentially an adaptation of existing code that 
was using non-blocking NIO low-level APIs directly, and it was already 
handling these issues, including potentialy partial length values.
The code for this can be found there:
http://jppf-project.cvs.sourceforge.net/viewvc/jppf-project/server/src/java/org/jppf/server/mina/nodeserver/NodeMessage.java?revision=1.1&view=markup
In any case, if I were missing one of these points, it wouldn't even be 
working with a single client, would it?

As per your request, I can put together a simple test that will 
reproduce the issue. I just need to take some time to package it, I hope 
it's OK if I send it tomorrow or the day after?

Thanks,
-Laurent


On 03/01/2010 13:53, Emmanuel Lecharny wrote:
Ok, after a quick check of your code, some other remarks :

- as you have no guarantee that you may even receive the length of 
your serialized object in one single message, you must absolutely deal 
with this problem by rying to read the four first bytes, storing them 
in the session if you didn't recived all of them (you may receive 3 
bytes, for instance).
- then when you have the length, you have to read the incoming length 
bytes from the buffer, and accumulate those bytes in the session if 
you expect some more to arrvie (ie, if the length is 2070 and you have 
received a 2048 size buffer, you have to wait for another message)
- *don't* try to deserialize the object until you have received all 
the needed bytes : it's a CPU waste !
- as you may have more than one object in your buffer, you have to 
iterate from step one until you either have no more bytes, or until 
you can't process the next object because you need more bytes. For 
each processed object, call the next filter.

Doing so should work.

Je peux aussi le faire en français si c'est plus clair, mais dans ce 
cas, le mieux, c'est de me pinger sur IRC (irc.freenode.net), channel 
#mina 
(http://mina.apache.org/contact.html#Contact-OtherWaystoContacttheMINACommunity) 


Bon dimanche !

Laurent Cohen a écrit :
Hi Ashish,

Thanks for the prompt reply.
Below is a typical track trace I'm getting. You'll notice it has 
nothing to do with Mina, as my clients do not use it yet.

java.lang.NegativeArraySizeException
  at 
org.jppf.comm.socket.AbstractSocketWrapper.receiveBytes(AbstractSocketWrapper.java:195) 

  at 
org.jppf.server.node.JPPFContainer.deserializeObjects(JPPFContainer.java:127) 

  at org.jppf.server.node.NodeIO.deserializeObjects(NodeIO.java:167)
  at org.jppf.server.node.NodeIO.readObjects(NodeIO.java:120)
  at org.jppf.server.node.NodeIO.readTask(NodeIO.java:90)
  at org.jppf.server.node.JPPFNode.perform(JPPFNode.java:179)
  at org.jppf.server.node.JPPFNode.run(JPPFNode.java:136)
  at org.jppf.node.NodeRunner.main(NodeRunner.java:114)
2010-01-03 10:23:16,227 
[ERROR][org.jppf.server.node.JPPFContainer.call(261)]: invalid type 
code: 20 [object index: 0]
java.io.StreamCorruptedException: invalid type code: 20
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1356)
  at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1667)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1323)
  at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
  at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
  at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) 

  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
  at java.util.HashMap.readObject(HashMap.java:1030)
  at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
  at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 

  at java.lang.reflect.Method.invoke(Method.java:597)
  at 
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
  at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
  at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) 

  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
  at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
  at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
  at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753) 

  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
  at 
org.jppf.utils.ObjectSerializerImpl.deserialize(ObjectSerializerImpl.java:123) 

  at 
org.jppf.utils.ObjectSerializerImpl.deserialize(ObjectSerializerImpl.java:95) 

  at 
org.jppf.server.node.JPPFContainer$ObjectReadTask.call(JPPFContainer.java:257) 

  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) 

  at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) 

  at java.lang.Thread.run(Thread.java:619)

The IoFilter code can be found at:
http://jppf-project.cvs.sourceforge.net/viewvc/jppf-project/server/src/java/org/jppf/server/mina/nodeserver/NodeIoFilter.java?revision=1.1&view=markup 

The IoHandler:
http://jppf-project.cvs.sourceforge.net/viewvc/jppf-project/server/src/java/org/jppf/server/mina/nodeserver/NodeIoHandler.java?revision=1.1&view=markup 

The code that creates and starts the IoProcessor:
http://jppf-project.cvs.sourceforge.net/viewvc/jppf-project/server/src/java/org/jppf/server/mina/nodeserver/MinaNodeServer.java?revision=1.1&view=markup 


Let me know if you need more information.

Sincerely,
-Laurent

On 03/01/2010 11:09, Ashish wrote:
  Can we get the exception trace?. Also, is it possible to share your
custom IoFilter?

thanks
ashish

On Sun, Jan 3, 2010 at 3:58 AM, Laurent 
Cohen<laurent.co...@jppf.org>  wrote:
Hello,

I've just started with Mina and am facing a problem I've been stuck 
with for
days.
Environment:
-  Mina 2.0.0 RC1
- Sun JDK 1.5 / 1.6 / 1.6 x64
- Windows Vista x64

I'm currently trying to switch an open source grid computing project
(http://www.jppf.org), of which I am a developer, to Mina's 
communication
framework, as I believe it would be easier to maintain and extend, 
and I'm
very excited by the built-in SSL and in-JVM communication 
capabilities.
The architecture of the project is of type master / worker where 
the master
acts as the Mina server, and the worker nodes as the clients.
Currently, I am at a point where the communication between server 
and client
is working, but only with a single client/node. As soon as I have 
more than
one client, exceptions are thrown on the clients' side on write 
operations.
Each message exchanged between server and client is made of a set of
serialized Java objects, with a header for each object which is a 
4-bytes
integer that indicates the length of the serialized block of data. 
The first
serialized object (i.e. message header) tells how many objects are 
in the
message.
Basically each message is as follows: [ length1, data1, ... , 
lengthN, dataN
] where lengthX is the length of block dataX, which is a serialized 
object
graph.

 From the tracing and debugging I've done, the first object is 
always read
and deserialized properly, and it is always on the second object 
that the
exceptions occur. The exceptions are of types 
StreamCorruptedException,
NegativeArraySizeException, OutOfMemoryError, apparently due to the 
fact
that the block length is not read or written properly.

On the server side, serialization/deserialization occurs only for 
the first
(header) object of the message. The other objects are actually 
handled as
arrays of bytes, but not necessarily in memory: some or all of them 
may be
stored in temporary files, depending on the remaining available 
memory in
the server's JVM. The goal is to allow the server to work with data
structures that are much larger than the JVM heap. This is why I 
implemented
a custom IoFilter, rather than for instance a codec like
CumulativeProtocolDecoder, as I do not want the entire message to 
be in
memory (only the header object, which is relatively small, has to be).

Sorry for the lengthy explanation, I hope it is clear enough. I am 
puzzled
as to why this is working with a single client, but not with 2 or 
more.
Would you have any idea as to where I can look, and as to what I 
could have
missed, overlooked, or done wrong?
Please let me know if you need more information or code snippets.

Sincerely,
-Laurent