Hi Vahram,
The socket leak was fixed in v1.6
https://issues.apache.org/jira/browse/GEODE-3563
On 1/30/19 7:12 AM, Vahram Aharonyan wrote:
Hi All,
At some circumstance we have seen following exception while using
geode-1.1.0 and trying to do Function Execution:
[severe 2018/05/22 01:32:31.964 EDT
Node-7ccee253-7f00-4670-960a-3ba6e5a2365b <Function Execution
Processor22> tid=0x58] ack read exception
org.apache.geode.ToDataException: toData failed on DataSerializable
class org.apache.geode.internal.cache.UpdateOperation$UpdateMessage
at
org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2436)
at
org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1446)
at
org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:232)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:377)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:237)
at
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:603)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1684)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875)
at
org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82)
at
org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416)
at
org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
at
org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:442)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621)
at
org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1067)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: An Exception was thrown
while serializing.
at
org.apache.geode.internal.tcp.MsgStreamer.writeAsSerializedByteArray(MsgStreamer.java:930)
at
org.apache.geode.DataSerializer.writeObjectAsByteArray(DataSerializer.java:1305)
at
org.apache.geode.internal.cache.DistributedCacheOperation.writeValue(DistributedCacheOperation.java:124)
at
org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:419)
at
org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2402)
... 61 more
Caused by: java.io.NotSerializableException: dataobject.PolicyInfo
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
At the same time, we have experienced OOM on our jvm and heapdump was
generated. Heapdump analysis showed that nearly half of the heap (16
GB of 32 GB) is consumed by SSLSocketImpl objects. Dominator Tree was
topped by 'Function Execution Processor22' thread, which is mentioned
above in the stacktrace including serialization exception.
So the main question is whether having NotSerializableException exception insendMessage() flow could result
to leak of not closed connections. Connection closing calls are spread over the code and is hard to track
them and I’m not sure If org.apache.geode.internal.tcp.MsgStreamer#closeis the one that is responsible to close connections in this case. But if so, connections will not be closed here as due to abovementioned NotSerializableException flushedByteswill not be updated in org.apache.geode.internal.tcp.MsgStreamer#writeMessage.
Can someone assist us to understand if this exception can yield to the
SSLSocketImpl leak or we should look into another direction?
Thanks,
Vahram.
