[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Ehm updated AMQ-4986:
---------------------------
Description:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=false+ failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
We do not know yet what triggers the wrong byte sequence from the client, but
on the brokers side, there should be a protection against this case.
was:
We experience this problem in combination with 5.5.1 client and the
wireformat.tightEncodingEnabled=true + failover:
Scenario:
1. start standard broker
2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
3. wait around 30sec (default for inactivity check)
Result:
The client closes the connection and re-tries to the broker which in turn
throws the following exception:
{code}
2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError: Java
heap space
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.OutOfMemoryError: Java heap space
at
org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
at
org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
... 1 more
{code}
The problem here is that the BaseDataStreamMarshaller reads an int from the
buffer and re-uses it immediately to allocate a byte array:
{code}
protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException {
byte rc[] = null;
if (dataIn.readBoolean()) {
int size = dataIn.readInt();
rc = new byte[size]; // PROBLEM! What happens if size has been
read and interpreted wrongly ?
dataIn.readFully(rc);
}
return rc;
}
{code}
In our case the dadaIn.readInt() read an int number of 785.477.224 which
triggers the broker to allocate blindly this amount of mem.
We do not know yet what triggers the wrong byte sequence from the client, but
on the brokers side, there should be a protection against this case.
> tightEncoding=false + failover triggers Broker memory exhaust
> -------------------------------------------------------------
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
> Reporter: Felix Ehm
> Priority: Critical
> Fix For: 5.10.0
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=true)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport: tcp:///10.18.3.97:60156@61616]
> DEBUG Transport Transport Connection to: tcp://10.18.3.97:60156 failed:
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
