[ 
https://issues.apache.org/jira/browse/ARTEMIS-739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508582#comment-15508582
 ] 

Damien Hollis commented on ARTEMIS-739:
---------------------------------------

In an attempt to try to track down the issue, I monitored files being opened 
and closed.  I noted that when large messages are moved to the DLQ, a copy is 
made (which involves opening the large message file) but I did not see the file 
associated with the copy being closed.

The stacktrace looks like this:
{noformat}
java.io.RandomAccessFile.open(String, int) RandomAccessFile.java
java.io.RandomAccessFile.<init>(File, String) RandomAccessFile.java:243
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(int, boolean) 
NIOSequentialFile.java:91
org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open() 
NIOSequentialFile.java:85
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.openFile()
 LargeServerMessageImpl.java:403
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.validateFile()
 LargeServerMessageImpl.java:380
org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.createLargeMessage(long,
 MessageInternal) JournalStorageManager.java:445
org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl.copy(long)
 LargeServerMessageImpl.java:278
org.apache.activemq.artemis.core.server.impl.ServerMessageImpl.makeCopyForExpiryOrDLA(long,
 MessageReference, boolean, boolean) ServerMessageImpl.java:214
org.apache.activemq.artemis.core.server.impl.QueueImpl.makeCopy(MessageReference,
 boolean, boolean) QueueImpl.java:2315
org.apache.activemq.artemis.core.server.impl.QueueImpl.makeCopy(MessageReference,
 boolean) QueueImpl.java:2297
org.apache.activemq.artemis.core.server.impl.QueueImpl.move(Transaction, 
SimpleString, MessageReference, boolean, boolean) QueueImpl.java:2381
org.apache.activemq.artemis.core.server.impl.QueueImpl.sendToDeadLetterAddress(Transaction,
 MessageReference, SimpleString) QueueImpl.java:2356
org.apache.activemq.artemis.core.server.impl.QueueImpl.checkRedelivery(MessageReference,
 long, boolean) QueueImpl.java:2124
org.apache.activemq.artemis.core.server.impl.RefsOperation.afterRollback(Transaction)
 RefsOperation.java:93
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterRollback(List)
 TransactionImpl.java:559
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.access$200(TransactionImpl,
 List) TransactionImpl.java:36
org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$4.done() 
TransactionImpl.java:404
org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run()
 OperationContextImpl.java:242
org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run()
 OrderedExecutorFactory.java:103
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 
ThreadPoolExecutor.java:1142
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:617
java.lang.Thread.run() Thread.java:745
{noformat}

It would seem that somewhere around the move we should closing or releasing the 
large message.
{noformat}
   private void move(final Transaction originalTX,
                     final SimpleString address,
                     final MessageReference ref,
                     final boolean rejectDuplicate,
                     final AckReason reason) throws Exception {
      Transaction tx;

      if (originalTX != null) {
         tx = originalTX;
      }
      else {
         // if no TX we create a new one to commit at the end
         tx = new TransactionImpl(storageManager);
      }

      ServerMessage copyMessage = makeCopy(ref, reason == AckReason.EXPIRED);

      copyMessage.setAddress(address);

      postOffice.route(copyMessage, null, tx, false, rejectDuplicate);

      acknowledge(tx, ref, reason);

      if (originalTX == null) {
         tx.commit();
      }
   }
{noformat}

I also noticed that LargeMessages implement finalize() and release their 
resources at that point but if the message is still sitting in a queue, it 
seems that an active reference to the message remains.

> Large messages failing with "(Too many open files)"
> ---------------------------------------------------
>
>                 Key: ARTEMIS-739
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-739
>             Project: ActiveMQ Artemis
>          Issue Type: Bug
>    Affects Versions: 1.3.0, 1.4.0
>            Reporter: Damien Hollis
>            Priority: Critical
>
> We are processing a large number of large messages and many of them are being 
> put into the dead-letter queue as part of a transaction rollback.  I suspect 
> that during the rollback the large message is not being released properly and 
> as a result the error below eventually occurs.  This may be a more general 
> issue but so far we have only seen this issue when processing messages 
> successfully (although another person in the team mentioned that there seem 
> to be a lot of large messages hanging around).
> I noted we are not using the latest version, so I'm in the process of 
> creating a new build and I will test with version 1.4 later today or tomorrow.
> {noformat}
> org.apache.activemq.artemis.core.server | AMQ222010: Critical IO Error, 
> shutting
> down the server. file=NIOSequentialFile 
> /var/data/artemis/large-messages/2147660860.msg, 
> message=/var/data/artemis/large-messages/2147660860.msg (Too many open files)
> org.apache.activemq.artemis.api.core.ActiveMQIOErrorException: 
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
>         at 
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:101)
>         at 
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:85)
>         at 
> org.apache.activemq.artemis.core.persistence.impl.journal.LargeServerMessageImpl$DecodingContext.open(LargeServerMessageImpl.java:426)
>         at 
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1131)
>         at 
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:414)
>         at 
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2464)
>         at 
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1956)
>         at 
> org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99)
>         at 
> org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2695)
>         at 
> org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: 
> /var/data/artemis/large-messages/2147660860.msg (Too many open files)
>         at java.io.RandomAccessFile.open0(Native Method)
>         at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
>         at 
> org.apache.activemq.artemis.core.io.nio.NIOSequentialFile.open(NIOSequentialFile.java:91)
>         ... 12 common frames omitted
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to