[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15533320#comment-15533320
]
ASF subversion and git services commented on ARTEMIS-739:
-
Commit 01503ed46e284579e38e101c6b509099c393e235 in activemq-artemis's branch
refs/heads/master from Clebert Suconic
[ https://git-wip-us.apache.org/repos/asf?p=activemq-artemis.git;h=01503ed ]
ARTEMIS-739 fixing checkstyle
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Assignee: Justin Bertram
>Priority: Critical
> Fix For: 1.5.0
>
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530442#comment-15530442
]
ASF subversion and git services commented on ARTEMIS-739:
-
Commit a6974596a0dc10cb48d66b301aff91d141a9bb7d in activemq-artemis's branch
refs/heads/master from jbertram
[ https://git-wip-us.apache.org/repos/asf?p=activemq-artemis.git;h=a697459 ]
ARTEMIS-739 fix large msg file leak on copy
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Assignee: Justin Bertram
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530444#comment-15530444
]
ASF GitHub Bot commented on ARTEMIS-739:
Github user asfgit closed the pull request at:
https://github.com/apache/activemq-artemis/pull/806
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Assignee: Justin Bertram
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530364#comment-15530364
]
ASF GitHub Bot commented on ARTEMIS-739:
Github user clebertsuconic commented on a diff in the pull request:
https://github.com/apache/activemq-artemis/pull/806#discussion_r80976998
--- Diff:
tests/integration-tests/src/test/java/org/apache/activemq/artemis/tests/integration/client/DeadLetterAddressTest.java
---
@@ -82,6 +87,43 @@ public void testBasicSend() throws Exception {
Assert.assertEquals(m.getBodyBuffer().readString(), "heyho!");
}
+ @Test
+ public void testLargeMessageFileLeak() throws Exception {
+ // only run this on *nix systems which will have the
com.sun.management.UnixOperatingSystemMXBean (needed to check open file count)
+ Assume.assumeTrue(File.separatorChar == '/');
--- End diff --
I tested it on windows and it does compile file...
Can you change this to some sort of instanceof?
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Assignee: Justin Bertram
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530175#comment-15530175
]
ASF GitHub Bot commented on ARTEMIS-739:
Github user jbertram commented on the issue:
https://github.com/apache/activemq-artemis/pull/806
No. I'm not sure it builds on Windows as I have no Windows machine to
build it on.
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Assignee: Justin Bertram
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530173#comment-15530173
]
ASF GitHub Bot commented on ARTEMIS-739:
Github user clebertsuconic commented on the issue:
https://github.com/apache/activemq-artemis/pull/806
are you sure this builds on windows?
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Assignee: Justin Bertram
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15527523#comment-15527523
]
ASF GitHub Bot commented on ARTEMIS-739:
GitHub user jbertram opened a pull request:
https://github.com/apache/activemq-artemis/pull/806
ARTEMIS-739 fix large msg file leak on copy
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jbertram/activemq-artemis ARTEMIS-739
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/activemq-artemis/pull/806.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #806
commit 726f99256ec495446e0d5bc7aab92a289362d1cd
Author: jbertram
Date: 2016-09-27T21:49:52Z
ARTEMIS-739 fix large msg file leak on copy
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Assignee: Justin Bertram
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508582#comment-15508582
]
Damien Hollis commented on ARTEMIS-739:
---
In an attempt to try to track down the issue, I monitored files being opened
and closed. I noted that when large messages are moved to the DLQ, a copy is
made (which involves opening the large message file) but I did not see the file
associated with the copy being closed.
The stacktrace looks like this:
{noformat}
java.io.RandomAccessFile.open(String, int) RandomAccessFile.java
java.io.RandomAccessFile.(File, String) RandomAccessFile.java:243
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(int, boolean)
NIOSequentialFile.java:91
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open()
NIOSequentialFile.java:85
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile()
LargeServerMessageImpl.java:403
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile()
LargeServerMessageImpl.java:380
org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(long,
MessageInternal) JournalStorageManager.java:445
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.copy(long)
LargeServerMessageImpl.java:278
org.apache.activemq.artemis.core.server.impl.ServerMessageImpl.makeCopyForExpiryOrDLA(long,
MessageReference, boolean, boolean) ServerMessageImpl.java:214
org.apache.activemq.artemis.core.server.impl.QueueImpl.makeCopy(MessageReference,
boolean, boolean) QueueImpl.java:2315
org.apache.activemq.artemis.core.server.impl.QueueImpl.makeCopy(MessageReference,
boolean) QueueImpl.java:2297
org.apache.activemq.artemis.core.server.impl.QueueImpl.move(Transaction,
SimpleString, MessageReference, boolean, boolean) QueueImpl.java:2381
org.apache.activemq.artemis.core.server.impl.QueueImpl.sendToDeadLetterAddress(Transaction,
MessageReference, SimpleString) QueueImpl.java:2356
org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(MessageReference,
long, boolean) QueueImpl.java:2124
org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(Transaction)
RefsOperation.java:93
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(List)
TransactionImpl.java:559
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.access$200(TransactionImpl,
List) TransactionImpl.java:36
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done()
TransactionImpl.java:404
org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run()
OperationContextImpl.java:242
org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run()
OrderedExecutorFactory.java:103
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
ThreadPoolExecutor.java:1142
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:617
java.lang.Thread.run() Thread.java:745
{noformat}
It would seem that somewhere around the move we should closing or releasing the
large message.
{noformat}
private void move(final Transaction originalTX,
final SimpleString address,
final MessageReference ref,
final boolean rejectDuplicate,
final AckReason reason) throws Exception {
Transaction tx;
if (originalTX != null) {
tx = originalTX;
}
else {
// if no TX we create a new one to commit at the end
tx = new TransactionImpl(storageManager);
}
ServerMessage copyMessage = makeCopy(ref, reason == AckReason.EXPIRED);
copyMessage.setAddress(address);
postOffice.route(copyMessage, null, tx, false, rejectDuplicate);
acknowledge(tx, ref, reason);
if (originalTX == null) {
tx.commit();
}
}
{noformat}
I also noticed that LargeMessages implement finalize() and release their
resources at that point but if the message is still sitting in a queue, it
seems that an active reference to the message remains.
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue
[jira] [Commented] (ARTEMIS-739) Large messages failing with "(Too many open files)"
[
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508336#comment-15508336
]
Damien Hollis commented on ARTEMIS-739:
---
Retested with 1.4.0 and the problem still exists.
> Large messages failing with "(Too many open files)"
> ---
>
> Key: ARTEMIS-739
> URL: https://issues.apache.org/jira/browse/ARTEMIS-739
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 1.3.0, 1.4.0
>Reporter: Damien Hollis
>Priority: Critical
>
> We are processing a large number of large messages and many of them are being
> put into the dead-letter queue as part of a transaction rollback. I suspect
> that during the rollback the large message is not being released properly and
> as a result the error below eventually occurs. This may be a more general
> issue but so far we have only seen this issue when processing messages
> successfully (although another person in the team mentioned that there seem
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error,
> shutting
> down the server. file=NIOSequentialFile
> /var/data/artemis/large-messages/2147660860.msg,
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
> at
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException:
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
> at java.io.RandomAccessFile.open0(Native Method)
> at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
> at java.io.RandomAccessFile.(RandomAccessFile.java:243)
> at
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
> ... 12 common frames omitted
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
