[
https://issues.apache.org/jira/browse/AMQ-4986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14260850#comment-14260850
]
siddeshwar_hiremath edited comment on AMQ-4986 at 12/30/14 6:30 AM:
--------------------------------------------------------------------
Broker version 5.10.0
Client version 5.10.0
activemq.xml:-
<transportConnector
uri="nio://localhost:61616?keepAlive=true&wireFormat.maxFrameSize=104857600"/>
Client broker url:-
failover:(nio://
127.0.0.1:61616?keepAlive=true&wireFormat.maxFrameSize=104857600)?jms.blobTransferPolicy.defaultUploadUrl=http://127.0.0.1:8161/fileserver/&initialReconnectDelay=10000&useExponentialBackOff=false&randomize=false
I have the above setting and experiencing following error:-
nio protocol:-
---------------
2014-12-29 11:52:43,145 | DEBUG | Sending: WireFormatInfo { version=10,
properties={MaxFrameSize=9223372036854775807, CacheSize=1024,
CacheEnabled=true, SizePrefixDisabled=false,
MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true,
MaxInactivityDuration=30000, TightEncodingEnabled=true,
StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} |
org.apache.activemq.transport.WireFormatNegotiator | ActiveMQ
BrokerService[localhost] Task-4
2014-12-29 11:52:43,591 | DEBUG | Transport Connection to:
tcp://127.0.0.1:51048 failed: java.io.IOException: Java heap space |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 5
java.io.IOException: Java heap space
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:151)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0.jar:5.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_40]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_40]
at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
2014-12-29 11:52:43,592 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=nio_//localhost_61616&qe;transport.keepAlive&true&transport.wireFormat.maxFrameSize&104857600,connectionViewType=remoteAddress,connectionName=tcp_//127.0.0.1_51048
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ NIO Worker 5
2014-12-29 11:52:43,605 | DEBUG | Stopping connection: tcp://127.0.0.1:51048 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
2014-12-29 11:52:43,609 | DEBUG | Stopping transport
tcp:///127.0.0.1:51048@61616 | org.apache.activemq.transport.tcp.TcpTransport |
ActiveMQ BrokerService[localhost] Task-5
2014-12-29 11:52:43,609 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task]
using ExecutorService: null | org.apache.activemq.thread.TaskRunnerFactory |
ActiveMQ BrokerService[localhost] Task-5
2014-12-29 11:52:43,614 | DEBUG | Closed socket Socket[unconnected] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2014-12-29 11:52:43,614 | DEBUG | Stopped transport: null |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
2014-12-29 11:52:43,614 | DEBUG | Connection Stopped: null |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
With tcp protocol:-
---------------------
ityDurationInitalDelay=10000, TcpNoDelayEnabled=true,
MaxInactivityDuration=30000, TightEncodingEnabled=true,
StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} |
org.apache.activemq.transport.WireFormatNegotiator | ActiveMQ
BrokerService[localhost] Task-8
2014-12-29 11:22:42,797 | DEBUG | Transport Connection to:
tcp://127.0.0.1:39382 failed: java.io.IOException: Unknown data type: 13 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport:
tcp:///127.0.0.1:39382@61616
java.io.IOException: Unknown data type: 13
at
org.apache.activemq.openwire.OpenWireFormat.looseUnmarshalNestedObject(OpenWireFormat.java:460)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.looseUnmarsalNestedObject(BaseDataStreamMarshaller.java:466)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.v10.DataResponseMarshaller.looseUnmarshal(DataResponseMarshaller.java:113)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:356)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)[activemq-client-5.10.0.jar:5.10.0]
at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
2014-12-29 11:22:42,799 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=tcp_//localhost_61616&qe;transport.keepAlive&true&transport.wireFormat.maxFrameSize&104857600,connectionViewType=remoteAddress,connectionName=tcp_//127.0.0.1_39382
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport:
tcp:///127.0.0.1:39382@61616
2014-12-29 11:22:42,804 | DEBUG | Stopping connection: tcp://127.0.0.1:39382 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-9
2014-12-29 11:22:42,805 | DEBUG | Stopping transport
tcp:///127.0.0.1:39382@61616 | org.apache.activemq.transport.tcp.TcpTransport |
ActiveMQ BrokerService[localhost] Task-9
2014-12-29 11:22:42,805 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task]
using ExecutorService: null | org.apache.activemq.thread.TaskRunnerFactory |
ActiveMQ BrokerService[localhost] Task-9
2014-12-29 11:22:42,807 | DEBUG | Closed socket
Socket[addr=/127.0.0.1,port=39382,localport=61616] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2014-12-29 11:22:42,808 | DEBUG | Stopped transport: tcp://127.0.0.1:39382 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-9
2014-12-29 11:22:42,809 | DEBUG | Connection Stopped: tcp://127.0.0.1:39382 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-9
was (Author: siddeshwar_hiremath):
Made following change:-
activemq.xml:-
connection.uri=nio://localhost:61616
<transportConnector
uri="${connection.uri}?keepAlive=true&wireFormat.maxFrameSize=104857600"/>
Client broker url:-
failover:(nio://
127.0.0.1:61616?keepAlive=true&wireFormat.maxFrameSize=104857600)?jms.blobTransferPolicy.defaultUploadUrl=http://127.0.0.1:8161/fileserver/&initialReconnectDelay=10000&useExponentialBackOff=false&randomize=false
I have the above setting and experiencing following error:-
nio protocol:-
---------------
2014-12-29 11:52:43,145 | DEBUG | Sending: WireFormatInfo { version=10,
properties={MaxFrameSize=9223372036854775807, CacheSize=1024,
CacheEnabled=true, SizePrefixDisabled=false,
MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true,
MaxInactivityDuration=30000, TightEncodingEnabled=true,
StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} |
org.apache.activemq.transport.WireFormatNegotiator | ActiveMQ
BrokerService[localhost] Task-4
2014-12-29 11:52:43,591 | DEBUG | Transport Connection to:
tcp://127.0.0.1:51048 failed: java.io.IOException: Java heap space |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 5
java.io.IOException: Java heap space
at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:151)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0.jar:5.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_40]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_40]
at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
2014-12-29 11:52:43,592 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=nio_//localhost_61616&qe;transport.keepAlive&true&transport.wireFormat.maxFrameSize&104857600,connectionViewType=remoteAddress,connectionName=tcp_//127.0.0.1_51048
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ NIO Worker 5
2014-12-29 11:52:43,605 | DEBUG | Stopping connection: tcp://127.0.0.1:51048 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
2014-12-29 11:52:43,609 | DEBUG | Stopping transport
tcp:///127.0.0.1:51048@61616 | org.apache.activemq.transport.tcp.TcpTransport |
ActiveMQ BrokerService[localhost] Task-5
2014-12-29 11:52:43,609 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task]
using ExecutorService: null | org.apache.activemq.thread.TaskRunnerFactory |
ActiveMQ BrokerService[localhost] Task-5
2014-12-29 11:52:43,614 | DEBUG | Closed socket Socket[unconnected] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2014-12-29 11:52:43,614 | DEBUG | Stopped transport: null |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
2014-12-29 11:52:43,614 | DEBUG | Connection Stopped: null |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-5
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
With tcp protocol:-
---------------------
ityDurationInitalDelay=10000, TcpNoDelayEnabled=true,
MaxInactivityDuration=30000, TightEncodingEnabled=true,
StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]} |
org.apache.activemq.transport.WireFormatNegotiator | ActiveMQ
BrokerService[localhost] Task-8
2014-12-29 11:22:42,797 | DEBUG | Transport Connection to:
tcp://127.0.0.1:39382 failed: java.io.IOException: Unknown data type: 13 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport:
tcp:///127.0.0.1:39382@61616
java.io.IOException: Unknown data type: 13
at
org.apache.activemq.openwire.OpenWireFormat.looseUnmarshalNestedObject(OpenWireFormat.java:460)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.looseUnmarsalNestedObject(BaseDataStreamMarshaller.java:466)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.v10.DataResponseMarshaller.looseUnmarshal(DataResponseMarshaller.java:113)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:356)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)[activemq-client-5.10.0.jar:5.10.0]
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)[activemq-client-5.10.0.jar:5.10.0]
at java.lang.Thread.run(Thread.java:724)[:1.7.0_40]
2014-12-29 11:22:42,799 | DEBUG | Unregistering MBean
org.apache.activemq:type=Broker,brokerName=localhost,connector=clientConnectors,connectorName=tcp_//localhost_61616&qe;transport.keepAlive&true&transport.wireFormat.maxFrameSize&104857600,connectionViewType=remoteAddress,connectionName=tcp_//127.0.0.1_39382
| org.apache.activemq.broker.jmx.ManagementContext | ActiveMQ Transport:
tcp:///127.0.0.1:39382@61616
2014-12-29 11:22:42,804 | DEBUG | Stopping connection: tcp://127.0.0.1:39382 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-9
2014-12-29 11:22:42,805 | DEBUG | Stopping transport
tcp:///127.0.0.1:39382@61616 | org.apache.activemq.transport.tcp.TcpTransport |
ActiveMQ BrokerService[localhost] Task-9
2014-12-29 11:22:42,805 | DEBUG | Initialized TaskRunnerFactory[ActiveMQ Task]
using ExecutorService: null | org.apache.activemq.thread.TaskRunnerFactory |
ActiveMQ BrokerService[localhost] Task-9
2014-12-29 11:22:42,807 | DEBUG | Closed socket
Socket[addr=/127.0.0.1,port=39382,localport=61616] |
org.apache.activemq.transport.tcp.TcpTransport | ActiveMQ Task-1
2014-12-29 11:22:42,808 | DEBUG | Stopped transport: tcp://127.0.0.1:39382 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-9
2014-12-29 11:22:42,809 | DEBUG | Connection Stopped: tcp://127.0.0.1:39382 |
org.apache.activemq.broker.TransportConnection | ActiveMQ
BrokerService[localhost] Task-9
> tightEncoding=false + failover triggers Broker memory exhaust
> -------------------------------------------------------------
>
> Key: AMQ-4986
> URL: https://issues.apache.org/jira/browse/AMQ-4986
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.6.0, 5.7.0, 5.8.0, 5.9.0
> Environment: java 1.7
> Client library 5.5.1
> Reporter: Felix Ehm
> Fix For: NEEDS_REVIEW
>
>
> We experience this problem in combination with 5.5.1 client and the
> wireformat.tightEncodingEnabled=false+ failover:
> Scenario:
> 1. start standard broker
> 2. start Client (with e.g. a MessageListener) with failover protocol: e.g.
> failover:(tcp://123.123.123.123:61616?wireformat.tightEncodingEnabled=false)
> 3. wait around 30sec (default for inactivity check)
> Result:
> The client closes the connection and re-tries to the broker which in turn
> throws the following exception:
> {code}
> 2014-01-21 20:12:49,568 [ActiveMQ Transport:
> tcp:///123.123.123.123:60156@61616] DEBUG Transport Transport Connection to:
> tcp://124.124.124.124:60156 failed: java.io.IOException: Unexpected error
> occured: java.lang.OutOfMemoryError: Java heap space
> java.io.IOException: Unexpected error occured: java.lang.OutOfMemoryError:
> Java heap space
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.activemq.openwire.v8.BaseDataStreamMarshaller.looseUnmarshalByteSequence(BaseDataStreamMarshaller.java:638)
> at
> org.apache.activemq.openwire.v8.WireFormatInfoMarshaller.looseUnmarshal(WireFormatInfoMarshaller.java:132)
> at
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:373)
> at
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:285)
> at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
> ... 1 more
> {code}
> The problem here is that the BaseDataStreamMarshaller reads an int from the
> buffer and re-uses it immediately to allocate a byte array:
> {code}
> protected byte[] looseUnmarshalByteArray(DataInput dataIn) throws IOException
> {
> byte rc[] = null;
> if (dataIn.readBoolean()) {
> int size = dataIn.readInt();
> rc = new byte[size]; // PROBLEM! What happens if size has been
> read and interpreted wrongly ?
> dataIn.readFully(rc);
> }
> return rc;
> }
> {code}
> In our case the dadaIn.readInt() read an int number of 785.477.224 which
> triggers the broker to allocate blindly this amount of mem.
> We do not know yet what triggers the wrong byte sequence from the client, but
> on the brokers side, there should be a protection against this case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
