[jira] [Updated] (ARTEMIS-2180) Host and broker runs out of memory when stopping a backup in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo Provido updated ARTEMIS-2180: - Attachment: server3-backup-broker.xml > Host and broker runs out of memory when stopping a backup in a cluster > -- > > Key: ARTEMIS-2180 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2180 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.6.3 >Reporter: Simon Chalmers >Priority: Critical > Attachments: hs_err_pid15534.log, server2-primary-broker.xml, > server3-backup-broker.xml > > > When running a live-backup cluster pair and stopping the backup, the live > broker starts leaking memory, runs out of memory and core dumps. > This occurs during a performance test when the broker is under load; the > slave has been running successfully but is then terminated whilst the broker > is still under load. > Core dump attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2180) Host and broker runs out of memory when stopping a backup in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791506#comment-16791506 ] Leo Provido commented on ARTEMIS-2180: -- Hi [~jbertram]. I've tried reproducing this and this is what I did: * I spun 3 c5.4xlarge in AWS, - using Artemis 2.6.3, I create a master broker on one server, a slave broker on another, and a test client. I attached my broker configs ( [^server2-primary-broker.xml] [^server3-backup-broker.xml] ) * I then ran our test on our test client, which sends messages with mean message size of 1,000,000 bytes and concurrency of 40 for a total duration of 600 seconds. * Half-way through the test, I invoked _artemis stop_ on the slave broker, and then noticed WARN messages logged by the master broker. Attached the artemis.log produced by the master broker. [^artemis.log], which was concluded by "[org.apache.activemq.artemis.core.server] AMQ222010: Critical IO Error, shutting down the server". I ran this test 3 times using a fresh stack of servers with interruptions (invoke _artemis stop_ on the slave broker) and got the same outcome. I also ran this test 3 times using a fresh stack of servers without interruptions and didn't encounter the issue. I've yet to try this on 2.6.4 and the latest build. > Host and broker runs out of memory when stopping a backup in a cluster > -- > > Key: ARTEMIS-2180 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2180 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.6.3 >Reporter: Simon Chalmers >Priority: Critical > Attachments: artemis.log, hs_err_pid15534.log, > server2-primary-broker.xml, server3-backup-broker.xml > > > When running a live-backup cluster pair and stopping the backup, the live > broker starts leaking memory, runs out of memory and core dumps. > This occurs during a performance test when the broker is under load; the > slave has been running successfully but is then terminated whilst the broker > is still under load. > Core dump attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2180) Host and broker runs out of memory when stopping a backup in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo Provido updated ARTEMIS-2180: - Attachment: artemis.log > Host and broker runs out of memory when stopping a backup in a cluster > -- > > Key: ARTEMIS-2180 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2180 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.6.3 >Reporter: Simon Chalmers >Priority: Critical > Attachments: artemis.log, hs_err_pid15534.log, > server2-primary-broker.xml, server3-backup-broker.xml > > > When running a live-backup cluster pair and stopping the backup, the live > broker starts leaking memory, runs out of memory and core dumps. > This occurs during a performance test when the broker is under load; the > slave has been running successfully but is then terminated whilst the broker > is still under load. > Core dump attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2180) Host and broker runs out of memory when stopping a backup in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo Provido updated ARTEMIS-2180: - Attachment: server2-primary-broker.xml > Host and broker runs out of memory when stopping a backup in a cluster > -- > > Key: ARTEMIS-2180 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2180 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.6.3 >Reporter: Simon Chalmers >Priority: Critical > Attachments: hs_err_pid15534.log, server2-primary-broker.xml, > server3-backup-broker.xml > > > When running a live-backup cluster pair and stopping the backup, the live > broker starts leaking memory, runs out of memory and core dumps. > This occurs during a performance test when the broker is under load; the > slave has been running successfully but is then terminated whilst the broker > is still under load. > Core dump attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2270) Artemis created my broker in a different directory
[
https://issues.apache.org/jira/browse/ARTEMIS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785205#comment-16785205
]
Leo Provido commented on ARTEMIS-2270:
--
I just tried that (*artemis create "C:\Program Files\BrokerTEST*) and it
worked. Thanks.
> Artemis created my broker in a different directory
> --
>
> Key: ARTEMIS-2270
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2270
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.6.4
>Reporter: Leo Provido
>Priority: Trivial
>
> After I type {{artemis create C:\Program Files\BrokerTEST}} in the command
> line while in {{C:\Program Files\apache-artemis-2.6.4\bin}} and do the user
> name, password and allow-anonymous settings, the creation process of the
> broker proceeds and Artemis says that:
> {noformat}
> You can now start the broker by executing:
>"C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis"
> run
> Or you can setup the broker as Windows service and run it in the background:
>"C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe"
> install
>"C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" start
>To stop the windows service:
> "C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" stop
>To uninstall the windows service
> "C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe"
> uninstall
> {noformat}
> Is there a particular reason why the Artemis app decided to create the broker
> in {{C:\Program Files\apache-artemis-2.6.4\bin\Files\}} instead of
> {{C:\Program Files\}}?
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (ARTEMIS-2270) Artemis created my broker in a different directory
[
https://issues.apache.org/jira/browse/ARTEMIS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785205#comment-16785205
]
Leo Provido edited comment on ARTEMIS-2270 at 3/6/19 4:07 AM:
--
I just tried that (*artemis create "C:\Program Files\BrokerTEST"*) and it
worked. Thanks.
was (Author: leoprovido):
I just tried that (*artemis create "C:\Program Files\BrokerTEST*) and it
worked. Thanks.
> Artemis created my broker in a different directory
> --
>
> Key: ARTEMIS-2270
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2270
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.6.4
>Reporter: Leo Provido
>Priority: Trivial
>
> After I type {{artemis create C:\Program Files\BrokerTEST}} in the command
> line while in {{C:\Program Files\apache-artemis-2.6.4\bin}} and do the user
> name, password and allow-anonymous settings, the creation process of the
> broker proceeds and Artemis says that:
> {noformat}
> You can now start the broker by executing:
>"C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis"
> run
> Or you can setup the broker as Windows service and run it in the background:
>"C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe"
> install
>"C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" start
>To stop the windows service:
> "C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" stop
>To uninstall the windows service
> "C:\Program
> Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe"
> uninstall
> {noformat}
> Is there a particular reason why the Artemis app decided to create the broker
> in {{C:\Program Files\apache-artemis-2.6.4\bin\Files\}} instead of
> {{C:\Program Files\}}?
> Thanks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2270) Artemis created my broker in a different directory
[ https://issues.apache.org/jira/browse/ARTEMIS-2270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo Provido updated ARTEMIS-2270: - Summary: Artemis created my broker in a different directory (was: Artemis created by broker in a different directory) > Artemis created my broker in a different directory > -- > > Key: ARTEMIS-2270 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2270 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 2.6.4 >Reporter: Leo Provido >Priority: Trivial > > After I type *artemis create C:\Program Files\BrokerTEST* in the command line > while in *C:\Program Files\apache-artemis-2.6.4\bin>* and do the user name, > password and allow-anonymous settings, the creation process of the broker > proceeds and Artemis says that: > *You can now start the broker by executing: >"C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis" > run > Or you can setup the broker as Windows service and run it in the background: >"C:\Program > Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" > install >"C:\Program > Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" start >To stop the windows service: > "C:\Program > Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" stop >To uninstall the windows service > "C:\Program > Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" > uninstall* > Is there a particular reason why the Artemis app decided to create the broker > in *C:\Program Files\apache-artemis-2.6.4\bin\Files\* instead of *C:\Program > Files\*? > Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-2270) Artemis created by broker in a different directory
Leo Provido created ARTEMIS-2270: Summary: Artemis created by broker in a different directory Key: ARTEMIS-2270 URL: https://issues.apache.org/jira/browse/ARTEMIS-2270 Project: ActiveMQ Artemis Issue Type: Bug Affects Versions: 2.6.4 Reporter: Leo Provido After I type *artemis create C:\Program Files\BrokerTEST* in the command line while in *C:\Program Files\apache-artemis-2.6.4\bin>* and do the user name, password and allow-anonymous settings, the creation process of the broker proceeds and Artemis says that: *You can now start the broker by executing: "C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis" run Or you can setup the broker as Windows service and run it in the background: "C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" install "C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" start To stop the windows service: "C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" stop To uninstall the windows service "C:\Program Files\apache-artemis-2.6.4\bin\Files\BrokerTEST\bin\artemis-service.exe" uninstall* Is there a particular reason why the Artemis app decided to create the broker in *C:\Program Files\apache-artemis-2.6.4\bin\Files\* instead of *C:\Program Files\*? Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2268) Encountered exception [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext] Expected Symbol type but found encoding: -95: org.apache.qpid.pro
[ https://issues.apache.org/jira/browse/ARTEMIS-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785176#comment-16785176 ] Leo Provido commented on ARTEMIS-2268: -- Hey [~gemmellr]. We have test code in C# that fails in this way every time with 2.7-SNAPSHOT and not with 2.6.3 or 2.6.4, but it depends on proprietary code, so unfortunately it’s not something we can share. If we go to the effort of boiling it down to a sample that uses pure AmqpNetLite, can you confirm that the Artemis team will compile the code, run it and confirm whether you can reproduce the issue? We ask because we have been told previously that we must produce a Java sample for action to be taken. Otherwise, we can produce an executable that you may run to reproduce the issue. We’re confident that anything that can cause Artemis to crash is a concern for everyone. > Encountered exception > [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext] > Expected Symbol type but found encoding: -95: > org.apache.qpid.proton.ProtonException: Expected Symbol type but found > encoding: -95 > --- > > Key: ARTEMIS-2268 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2268 > Project: ActiveMQ Artemis > Issue Type: Bug >Reporter: Leo Provido >Priority: Major > > We were trying out some existing tests with the latest build of Artemis > (2.7.0-SNAPSHOT - as of 27-Feb-2019) and we encountered an exception thrown. > When this occurs, our consumer is replying to a received message (i.e. > sending a message to another queue on the same broker). The exception > message that is shown in the Artemis log is also reported back via the AMQP > client library. > Below shows the exception and stack trace found in artemis-service.out.log. > 2019-03-05 16:17:44,954 WARN > [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext] > Expected Symbol type but found encoding: -95: > org.apache.qpid.proton.ProtonException: Expected Symbol type but found > encoding: -95 > at > org.apache.qpid.proton.codec.DecoderImpl.readSymbol(DecoderImpl.java:771) > [proton-j-0.31.0.jar:] > at > org.apache.qpid.proton.codec.messaging.FastPathMessageAnnotationsType.readValue(FastPathMessageAnnotationsType.java:126) > [proton-j-0.31.0.jar:] > at > org.apache.qpid.proton.codec.messaging.FastPathMessageAnnotationsType.readValue(FastPathMessageAnnotationsType.java:43) > [proton-j-0.31.0.jar:] > at > org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.scanMessageData(AMQPMessage.java:480) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.ensureMessageDataScanned(AMQPMessage.java:437) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.(AMQPMessage.java:177) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:433) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext.actualDelivery(ProtonServerReceiverContext.java:304) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext.onMessage(ProtonServerReceiverContext.java:299) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:541) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:484) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:284) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:241) > [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] > at > org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:170) >
[jira] [Created] (ARTEMIS-2268) Encountered exception [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext] Expected Symbol type but found encoding: -95: org.apache.qpid.proto
Leo Provido created ARTEMIS-2268: Summary: Encountered exception [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext] Expected Symbol type but found encoding: -95: org.apache.qpid.proton.ProtonException: Expected Symbol type but found encoding: -95 Key: ARTEMIS-2268 URL: https://issues.apache.org/jira/browse/ARTEMIS-2268 Project: ActiveMQ Artemis Issue Type: Bug Reporter: Leo Provido We were trying out some existing tests with the latest build of Artemis (2.7.0-SNAPSHOT - as of 27-Feb-2019) and we encountered an exception thrown. When this occurs, our consumer is replying to a received message (i.e. sending a message to another queue on the same broker). The exception message that is shown in the Artemis log is also reported back via the AMQP client library. Below shows the exception and stack trace found in artemis-service.out.log. 2019-03-05 16:17:44,954 WARN [org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext] Expected Symbol type but found encoding: -95: org.apache.qpid.proton.ProtonException: Expected Symbol type but found encoding: -95 at org.apache.qpid.proton.codec.DecoderImpl.readSymbol(DecoderImpl.java:771) [proton-j-0.31.0.jar:] at org.apache.qpid.proton.codec.messaging.FastPathMessageAnnotationsType.readValue(FastPathMessageAnnotationsType.java:126) [proton-j-0.31.0.jar:] at org.apache.qpid.proton.codec.messaging.FastPathMessageAnnotationsType.readValue(FastPathMessageAnnotationsType.java:43) [proton-j-0.31.0.jar:] at org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.scanMessageData(AMQPMessage.java:480) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.ensureMessageDataScanned(AMQPMessage.java:437) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage.(AMQPMessage.java:177) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.serverSend(AMQPSessionCallback.java:433) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext.actualDelivery(ProtonServerReceiverContext.java:304) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerReceiverContext.onMessage(ProtonServerReceiverContext.java:299) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:541) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:484) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:284) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:241) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:170) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:149) [artemis-amqp-protocol-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:643) [artemis-server-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73) [artemis-core-client-2.7.0-SNAPSHOT.jar:2.7.0-SNAPSHOT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-all-4.1.28.Final.jar:4.1.28.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-all-4.1.28.Final.jar:4.1.28.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-all-4.1.28.Final.jar:4.1.28.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
[jira] [Commented] (ARTEMIS-2247) Value argument in artermis-service.xml is incorrect
[
https://issues.apache.org/jira/browse/ARTEMIS-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770721#comment-16770721
]
Leo Provido commented on ARTEMIS-2247:
--
Thanks [~jbertram] and others. Cheers.
> Value argument in artermis-service.xml is incorrect
> ---
>
> Key: ARTEMIS-2247
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2247
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.6.4
>Reporter: Leo Provido
>Assignee: Justin Bertram
>Priority: Major
> Fix For: 2.7.0, 2.6.5
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When we tried starting a broker we created we observed that the broker failed
> to start.
> According to the "Using the Server"
> (https://activemq.apache.org/artemis/docs/latest/using-server.html)
> documentation, we simply have to do:
> {noformat}
> "/user/server/bin/artemis-service" start
> {noformat}
> We observed that a log file {{./log/artermis-service.err.log}} was generated
> after doing so, containing an error:
> {noformat}
> Error: Could not find or load main class Files\Artemis\etc
> {noformat}
> We later figured out that,
> {noformat}
> -Dartemis.instance.etc="%ARTEMIS_INSTANCE_ETC%"
> {noformat}
> Should be (without {{"}}):
> {noformat}
> -Dartemis.instance.etc=%ARTEMIS_INSTANCE_ETC%
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2246) Documentation for configuration max-disk-usage is inaccurate.
[ https://issues.apache.org/jira/browse/ARTEMIS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770723#comment-16770723 ] Leo Provido commented on ARTEMIS-2246: -- Thanks [~jbertram] and others. Cheers. > Documentation for configuration max-disk-usage is inaccurate. > - > > Key: ARTEMIS-2246 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2246 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 2.6.4 >Reporter: Leo Provido >Assignee: Justin Bertram >Priority: Major > Fix For: 2.7.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > During the execution of tests involving sending messages to the System, we > observed that the System started blocking messages before the disk is full. > According to the Configuration Reference > ([https://activemq.apache.org/artemis/docs/1.4.0/configuration-index.html]) > documentation, the default value of the configuration max-disk-usage is 100. > However, we discovered that the actual default value of max-disk-usage is 90. > The file inspected is ./etc/broker.xml. Thus, the documentation is deemed > inaccurate. > We have the opinion that severity of this incident is Major because a) the > documentation provides an inaccurate piece of information that influences the > way we design our mitigation strategies for risks surrounding hardware > resources utilization, and b) we don’t necessarily have the bandwidth to > review all default configuration values. > Inaccurate information may cause failure to provide message brokering > services despite mitigation strategies being in place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2172) Broker drops connection during throughput test
[
https://issues.apache.org/jira/browse/ARTEMIS-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770724#comment-16770724
]
Leo Provido commented on ARTEMIS-2172:
--
Hi [~jbertram]. Would you be able to assist on this particular issue? Thanks.
> Broker drops connection during throughput test
> --
>
> Key: ARTEMIS-2172
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2172
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.7.0, 2.6.3
> Environment: * Host OS: Linux node1 4.9.0-8-amd64 #1 SMP Debian
> 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux
> * Dotnet2.1 SDK installed as per these instructions:
> https://www.microsoft.com/net/download/linux-package-manager/debian9/sdk-current
> * Test client attached, to run the test: dotnet AmqpBrokerThroughputTest.dll
> -h -u -p , e.g. dotnet AmqpBrokerThroughputTest.dll
> -h 172.31.0.175 -u mimqadmin -p dogcat123
> * Broker.xml attached.
>Reporter: Simon Chalmers
>Priority: Major
> Attachments: AmqpBrokerThroughputTest.zip, broker.xml
>
>
> When running a continual test against a broker setup as a master-slave, after
> an arbitrary length of time, the test client disconnects and stops working.
> The only message thrown in the artemis.log file is:
> {{2018-10-31 09:32:07,368 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure has been detected: AMQ229014: Did not receive
> data from /172.31.15.214:38325 within the 60,000ms connection TTL. The
> connection will now be closed. [code=CONNECTION_TIMEDOUT]
> 2018-10-31 09:32:07,370 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 4610a425-dcbf-11e8-b83f-0284a4757140
> 2018-10-31 09:32:07,371 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 4610a425-dcbf-11e8-b83f-0284a4757140}}
> I wondered if this issue (https://issues.apache.org/jira/browse/ARTEMIS-2146)
> was related, but I tried the test against 2.6.x and the same result happened.
> I've tried it against the current version 2.6.3 and the 2.6.x master in
> github.
> Question: Is there more information available to see why a connection failed?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARTEMIS-2253) Creating a broker instance in "Program Files", we needed to specify "Progra~1" instead of "Program Files" in the directory parameter within a powershell script.
Leo Provido created ARTEMIS-2253: Summary: Creating a broker instance in "Program Files", we needed to specify "Progra~1" instead of "Program Files" in the directory parameter within a powershell script. Key: ARTEMIS-2253 URL: https://issues.apache.org/jira/browse/ARTEMIS-2253 Project: ActiveMQ Artemis Issue Type: Bug Affects Versions: 2.6.4 Reporter: Leo Provido When we tried creating a broker instance in "Program Files", we needed to specify "Progra~1" instead of "Program Files" in the directory parameter. According to the "Using the Server" (https://activemq.apache.org/artemis/docs/latest/using-server.html) documentation, we simply have to do: "artemis create " Note: We're running this command on a powershell script. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2252) Intermittent thread deadlock when using message groups and message selectors
[ https://issues.apache.org/jira/browse/ARTEMIS-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo Provido updated ARTEMIS-2252: - Summary: Intermittent thread deadlock when using message groups and message selectors (was: Some messages are never delivered.) > Intermittent thread deadlock when using message groups and message selectors > > > Key: ARTEMIS-2252 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2252 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.4 >Reporter: Leo Provido >Priority: Major > Attachments: Artemis 2.6.4 thread dump.txt > > > We’ve encountered what we think is a bug in Artemis. The issue arises > establishing two AMQP receiver links to the same Artemis queue (durable), > where each receiver link is established with an AMQP “filter set”, and then > messages are sent to the queue where those messages are assigned to message > groups. The messages are filtered on message type (JMSType). With this setup, > we occasionally find that Artemis permanently holds onto a message (i.e. > never delivers it). > We send 100,000 messages (of the same type – all messages go to the same > receiver link) to the queue, where each message is assigned to one of 10,000 > message groups, where each message group contains 10 messages. Furthermore, > the messages are assigned to message groups in the order such that: > • After the first 10,000 messages have been sent, each message group has > been assigned one message > • After the second 10,000 messages have been sent, each message group has > been assigned two messages > • After 90,000 message have been sent, each message group has been > assigned 9 messages. > The client application creating the two receiver links accepts all messages > in a message group when the last message in that message group is received. > We find the client application sometimes receives fewer than 100,000 messages > from Artemis. Around 1 in 3 the application receives 99,999 messages, but > sometimes, even fewer than 99,999 messages. After 2 minutes, we time out the > message group containing the one missing message and reject the 9 messages we > did receive. This then leaves one message in the queue. > This message is never delivered. However, more interestingly, when we then > attempt to close the connection, it takes 60 seconds to close. Furthermore, > after 30 seconds, Artemis logs a thread dump. After another 30 seconds, > Artemis logs a second thread dump, and then the connection is closed. The > attached is a copy of both thread dumps. > Note that if the second receiver link is not created, the issue does not > occur. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2252) Some messages are never delivered.
[ https://issues.apache.org/jira/browse/ARTEMIS-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo Provido updated ARTEMIS-2252: - Description: We’ve encountered what we think is a bug in Artemis. The issue arises establishing two AMQP receiver links to the same Artemis queue (durable), where each receiver link is established with an AMQP “filter set”, and then messages are sent to the queue where those messages are assigned to message groups. The messages are filtered on message type (JMSType). With this setup, we occasionally find that Artemis permanently holds onto a message (i.e. never delivers it). We send 100,000 messages (of the same type – all messages go to the same receiver link) to the queue, where each message is assigned to one of 10,000 message groups, where each message group contains 10 messages. Furthermore, the messages are assigned to message groups in the order such that: • After the first 10,000 messages have been sent, each message group has been assigned one message • After the second 10,000 messages have been sent, each message group has been assigned two messages • After 90,000 message have been sent, each message group has been assigned 9 messages. The client application creating the two receiver links accepts all messages in a message group when the last message in that message group is received. We find the client application sometimes receives fewer than 100,000 messages from Artemis. Around 1 in 3 the application receives 99,999 messages, but sometimes, even fewer than 99,999 messages. After 2 minutes, we time out the message group containing the one missing message and reject the 9 messages we did receive. This then leaves one message in the queue. This message is never delivered. However, more interestingly, when we then attempt to close the connection, it takes 60 seconds to close. Furthermore, after 30 seconds, Artemis logs a thread dump. After another 30 seconds, Artemis logs a second thread dump, and then the connection is closed. The attached is a copy of both thread dumps. Note that if the second receiver link is not created, the issue does not occur. was: We’ve encountered what we think is a bug in Artemis. The issue arises establishing two AMQP receiver links to the same Artemis queue (durable), where each receiver link is established with an AMQP “filter set”, and then messages are sent to the queue where those messages are assigned to message groups. The messages are filtered on message type (JMSType). With this setup, we occasionally find that Artemis permanently holds onto a message (i.e. never delivers it). We send 100,000 messages (of the same type – all messages go to the same receiver link) to the queue, where each message is assigned to one of 10,000 message groups, where each message group contains 10 messages. Furthermore, the messages are assigned to message groups in the order such that: • After the first 10,000 messages have been sent, each message group has been assigned one message • After the second 10,000 messages have been sent, each message group has been assigned two messages • After 90,000 message have been sent, each message group has been assigned 9 messages. The client application creating the two receiver links accepts all messages in a message group when the last message in that message group is received. We find the client application sometimes receives fewer than 100,000 messages from Artemis. Around 1 in 3 the application receives 99,999 messages, but sometimes, even fewer than 99,999 messages. After 2 minutes, we time out the message group containing the one missing message and reject the 9 messages we did receive. This then leaves one message in the queue. This message is never delivered. However, more interestingly, when we then attempt to close the connection, it takes 60 seconds to close. Furthermore, after 30 seconds, Artemis logs a thread dump. After another 30 seconds, Artemis logs a second thread dump, and then the connection is closed. The attached is a copy of both thread dumps. > Some messages are never delivered. > -- > > Key: ARTEMIS-2252 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2252 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: Broker >Affects Versions: 2.6.4 >Reporter: Leo Provido >Priority: Major > Attachments: Artemis 2.6.4 thread dump.txt > > > We’ve encountered what we think is a bug in Artemis. The issue arises > establishing two AMQP receiver links to the same Artemis queue (durable), > where each receiver link is established with an AMQP “filter set”, and then > messages are sent to the queue where those messages are assigned to message > groups. The messages are filtered on message type
[jira] [Created] (ARTEMIS-2252) Some messages are never delivered.
Leo Provido created ARTEMIS-2252: Summary: Some messages are never delivered. Key: ARTEMIS-2252 URL: https://issues.apache.org/jira/browse/ARTEMIS-2252 Project: ActiveMQ Artemis Issue Type: Bug Components: Broker Affects Versions: 2.6.4 Reporter: Leo Provido Attachments: Artemis 2.6.4 thread dump.txt We’ve encountered what we think is a bug in Artemis. The issue arises establishing two AMQP receiver links to the same Artemis queue (durable), where each receiver link is established with an AMQP “filter set”, and then messages are sent to the queue where those messages are assigned to message groups. The messages are filtered on message type (JMSType). With this setup, we occasionally find that Artemis permanently holds onto a message (i.e. never delivers it). We send 100,000 messages (of the same type – all messages go to the same receiver link) to the queue, where each message is assigned to one of 10,000 message groups, where each message group contains 10 messages. Furthermore, the messages are assigned to message groups in the order such that: • After the first 10,000 messages have been sent, each message group has been assigned one message • After the second 10,000 messages have been sent, each message group has been assigned two messages • After 90,000 message have been sent, each message group has been assigned 9 messages. The client application creating the two receiver links accepts all messages in a message group when the last message in that message group is received. We find the client application sometimes receives fewer than 100,000 messages from Artemis. Around 1 in 3 the application receives 99,999 messages, but sometimes, even fewer than 99,999 messages. After 2 minutes, we time out the message group containing the one missing message and reject the 9 messages we did receive. This then leaves one message in the queue. This message is never delivered. However, more interestingly, when we then attempt to close the connection, it takes 60 seconds to close. Furthermore, after 30 seconds, Artemis logs a thread dump. After another 30 seconds, Artemis logs a second thread dump, and then the connection is closed. The attached is a copy of both thread dumps. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2180) Host and broker runs out of memory when stopping a backup in a cluster
[ https://issues.apache.org/jira/browse/ARTEMIS-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764608#comment-16764608 ] Leo Provido commented on ARTEMIS-2180: -- Hi [~jbertram]. I'll be looking after this going forward. I'll find out what the actual test case was, reproduce the OOM error on 2.6.3 and then run the same test case on 2.6.4. I'll get back to you on the outcome. > Host and broker runs out of memory when stopping a backup in a cluster > -- > > Key: ARTEMIS-2180 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2180 > Project: ActiveMQ Artemis > Issue Type: Bug > Components: AMQP >Affects Versions: 2.6.3 >Reporter: Simon Chalmers >Priority: Critical > Attachments: hs_err_pid15534.log > > > When running a live-backup cluster pair and stopping the backup, the live > broker starts leaking memory, runs out of memory and core dumps. > This occurs during a performance test when the broker is under load; the > slave has been running successfully but is then terminated whilst the broker > is still under load. > Core dump attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-2172) Broker drops connection during throughput test
[
https://issues.apache.org/jira/browse/ARTEMIS-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762288#comment-16762288
]
Leo Provido commented on ARTEMIS-2172:
--
Hi [~clebertsuconic]. I’ll be looking after this from today. I looked at some
of the details of this bug report and I have a few clarifications on your last
comment so I can understand this better.
1. By “the client couldn't keep up with sending disconnects”, did you mean
the client cannot keep up receiving “disconnects” from the Artemis server?
Please provide more information about these “disconnects” that the Artemis
server is supposedly sending and what causes it to do so during the throughput
test.
2. Regarding the workaround you are proposing, can you provide more
information as to what you meant by “disable TTL” and the purpose of doing so
to address the particular issue that has occurred during the throughput test?
> Broker drops connection during throughput test
> --
>
> Key: ARTEMIS-2172
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2172
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: AMQP
>Affects Versions: 2.7.0, 2.6.3
> Environment: * Host OS: Linux node1 4.9.0-8-amd64 #1 SMP Debian
> 4.9.110-3+deb9u6 (2018-10-08) x86_64 GNU/Linux
> * Dotnet2.1 SDK installed as per these instructions:
> https://www.microsoft.com/net/download/linux-package-manager/debian9/sdk-current
> * Test client attached, to run the test: dotnet AmqpBrokerThroughputTest.dll
> -h -u -p , e.g. dotnet AmqpBrokerThroughputTest.dll
> -h 172.31.0.175 -u mimqadmin -p dogcat123
> * Broker.xml attached.
>Reporter: Simon Chalmers
>Priority: Major
> Attachments: AmqpBrokerThroughputTest.zip, broker.xml
>
>
> When running a continual test against a broker setup as a master-slave, after
> an arbitrary length of time, the test client disconnects and stops working.
> The only message thrown in the artemis.log file is:
> {{2018-10-31 09:32:07,368 WARN [org.apache.activemq.artemis.core.client]
> AMQ212037: Connection failure has been detected: AMQ229014: Did not receive
> data from /172.31.15.214:38325 within the 60,000ms connection TTL. The
> connection will now be closed. [code=CONNECTION_TIMEDOUT]
> 2018-10-31 09:32:07,370 WARN [org.apache.activemq.artemis.core.server]
> AMQ222061: Client connection failed, clearing up resources for session
> 4610a425-dcbf-11e8-b83f-0284a4757140
> 2018-10-31 09:32:07,371 WARN [org.apache.activemq.artemis.core.server]
> AMQ222107: Cleared up resources for session
> 4610a425-dcbf-11e8-b83f-0284a4757140}}
> I wondered if this issue (https://issues.apache.org/jira/browse/ARTEMIS-2146)
> was related, but I tried the test against 2.6.x and the same result happened.
> I've tried it against the current version 2.6.3 and the 2.6.x master in
> github.
> Question: Is there more information available to see why a connection failed?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ARTEMIS-2247) Value of an argument node related to ./bin/artermis-service.xml is incorrect
[ https://issues.apache.org/jira/browse/ARTEMIS-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leo Provido updated ARTEMIS-2247: - Summary: Value of an argument node related to ./bin/artermis-service.xml is incorrect (was: Value of an argument node related in ./bin/artermis-service.xml is incorrect) > Value of an argument node related to ./bin/artermis-service.xml is incorrect > > > Key: ARTEMIS-2247 > URL: https://issues.apache.org/jira/browse/ARTEMIS-2247 > Project: ActiveMQ Artemis > Issue Type: Bug >Affects Versions: 2.6.4 >Reporter: Leo Provido >Priority: Major > > When we tried When we tried starting a broker we created, we observed that > the broker failed to start.starting a broker we created, we observed that the > broker failed to start. > According to the Using the Server > (https://activemq.apache.org/artemis/docs/latest/using-server.html) > documentation, we simply have to do: > _Or you can run the broker in the background using: >"/user/server/bin/artemis-service" start_ > We observed that a log file ./log/artermis-service.err.log was generated > after doing so, containing an error Error: Could not find or load main class > Files\Artemis\etc > We later figured out that, > _-Dartemis.instance.etc="%ARTEMIS_INSTANCE_ETC%"_ > Should be (without “): > -Dartemis.instance.etc=%ARTEMIS_INSTANCE_ETC% > We have the opinion that the severity of this incident is High because this > issue prevents running the broker without fixing it. > An incorrect argument in ./bin/artermis-service.xml causes failure to run a > broker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-2247) Value of an argument node related in ./bin/artermis-service.xml is incorrect
Leo Provido created ARTEMIS-2247: Summary: Value of an argument node related in ./bin/artermis-service.xml is incorrect Key: ARTEMIS-2247 URL: https://issues.apache.org/jira/browse/ARTEMIS-2247 Project: ActiveMQ Artemis Issue Type: Bug Affects Versions: 2.6.4 Reporter: Leo Provido When we tried When we tried starting a broker we created, we observed that the broker failed to start.starting a broker we created, we observed that the broker failed to start. According to the Using the Server (https://activemq.apache.org/artemis/docs/latest/using-server.html) documentation, we simply have to do: _Or you can run the broker in the background using: "/user/server/bin/artemis-service" start_ We observed that a log file ./log/artermis-service.err.log was generated after doing so, containing an error Error: Could not find or load main class Files\Artemis\etc We later figured out that, _-Dartemis.instance.etc="%ARTEMIS_INSTANCE_ETC%"_ Should be (without “): -Dartemis.instance.etc=%ARTEMIS_INSTANCE_ETC% We have the opinion that the severity of this incident is High because this issue prevents running the broker without fixing it. An incorrect argument in ./bin/artermis-service.xml causes failure to run a broker. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARTEMIS-2246) Documentation for configuration max-disk-usage is inaccurate.
Leo Provido created ARTEMIS-2246: Summary: Documentation for configuration max-disk-usage is inaccurate. Key: ARTEMIS-2246 URL: https://issues.apache.org/jira/browse/ARTEMIS-2246 Project: ActiveMQ Artemis Issue Type: Bug Affects Versions: 2.6.4 Reporter: Leo Provido During the execution of tests involving sending messages to the System, we observed that the System started blocking messages before the disk is full. According to the Configuration Reference ([https://activemq.apache.org/artemis/docs/1.4.0/configuration-index.html]) documentation, the default value of the configuration max-disk-usage is 100. However, we discovered that the actual default value of max-disk-usage is 90. The file inspected is ./etc/broker.xml. Thus, the documentation is deemed inaccurate. We have the opinion that severity of this incident is Major because a) the documentation provides an inaccurate piece of information that influences the way we design our mitigation strategies for risks surrounding hardware resources utilization, and b) we don’t necessarily have the bandwidth to review all default configuration values. Inaccurate information may cause failure to provide message brokering services despite mitigation strategies being in place. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
