[
https://issues.apache.org/jira/browse/JAMES-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Tellier updated JAMES-3891:
----------------------------------
Description:
Given a mail queue currently containing emails and being consumed, When I
restart James, Then after the restart the email consumptions sometime do not
resume.
The following log can be seen:
{code:java}
james1 | java.io.FileNotFoundException:
/root/var/store/activemq/blob-transfer/10/ID_james1.local-33677-1677150190846-5_1_1_1_1037
(No such file or directory)
james1 | at java.base/java.io.FileInputStream.open0(Native Method)
james1 | at java.base/java.io.FileInputStream.open(Unknown Source)
james1 | at java.base/java.io.FileInputStream.<init>(Unknown
Source)
james1 | at
org.apache.james.queue.activemq.FileSystemBlobStrategy.getInputStream(FileSystemBlobStrategy.java:101)
james1 | at
org.apache.activemq.blob.BlobDownloader.getInputStream(BlobDownloader.java:38)
james1 | at
org.apache.activemq.command.ActiveMQBlobMessage.getInputStream(ActiveMQBlobMessage.java:132)
james1 | at
org.apache.james.queue.activemq.MimeMessageBlobMessageSource.getInputStream(MimeMessageBlobMessageSource.java:45)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:307)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:296)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:292)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:276)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.<init>(MimeMessageWrapper.java:158)
james1 | ... 19 common frames omitted
james1 | Wrapped by: javax.mail.MessagingException: IOException while
copying message
james1 | at
org.apache.james.server.core.MimeMessageWrapper.<init>(MimeMessageWrapper.java:165)
james1 | at
org.apache.james.server.core.MailImpl.setMessage(MailImpl.java:505)
james1 | at
com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73)
james1 | at java.base/java.util.Optional.ifPresent(Unknown Source)
james1 | at
org.apache.james.server.core.MailImpl$Builder.build(MailImpl.java:279)
james1 | at
org.apache.james.queue.jms.JMSCacheableMailQueue.createMail(JMSCacheableMailQueue.java:391)
james1 | at
org.apache.james.queue.activemq.ActiveMQCacheableMailQueue.createMailQueueItem(ActiveMQCacheableMailQueue.java:230)
james1 | at
org.apache.james.queue.jms.JMSCacheableMailQueue.deQueueOneItem(JMSCacheableMailQueue.java:249)
james1 | ... 12 common frames omitted
james1 | Wrapped by:
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue next
message
james1 | at
org.apache.james.queue.jms.JMSCacheableMailQueue.deQueueOneItem(JMSCacheableMailQueue.java:260)
james1 | at
reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
james1 | at
reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.resubscribe(FluxRepeatPredicate.java:120)
james1 | at
reactor.core.publisher.MonoRepeatPredicate.subscribeOrReturn(MonoRepeatPredicate.java:48)
james1 | at
reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:55)
james1 | at
reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
james1 | at
reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
james1 | at
reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
james1 | at java.base/java.util.concurrent.FutureTask.run(Unknown
Source)
james1 | at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
james1 | at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
james1 | at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
james1 | at java.base/java.lang.Thread.run(Unknown Source)
james1 | Wrapped by:
reactor.core.Exceptions$ErrorCallbackNotImplemented:
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue next
message;
james1 | nested exception is:
james1 | javax.mail.MessagingException: IOException while copying
message;
james1 | nested exception is:
james1 | java.io.FileNotFoundException:
/root/var/store/activemq/blob-transfer/10/ID_james1.local-33677-1677150190846-5_1_1_1_1037
(No such file or directory)
{code}
No processing takes place and remaining emails are stuck into the mail queue.
The issue is that blobs are handle via file so their deletion is not linked to
the JMS trasaction: the JMS transaction can be rolled back for a deleted blob,
thus leading to messages without blob.
Steps to reproduce:
- Send 5000 mails to james of 500 bytes each
{code:java}
smtp-source -A -C100 -r 1 -l 500 -m 5000 -s 20 -d -c -f [email protected] -M
other.com -t bob@localhost 172.24.0.4:25
{code}
- Once all mails are queued, restart james several time during processing
- Increasing the count of spooler workers helps.
As a fix I propose to discard emails not backed by a blob.
was:
Given a mail queue currently containing emails and being consumed, When I
restart James, Then after the restart the email consumptions sometime do not
resume.
The following log can be seen:
{code:java}
james1 | java.io.FileNotFoundException:
/root/var/store/activemq/blob-transfer/10/ID_james1.local-33677-1677150190846-5_1_1_1_1037
(No such file or directory)
james1 | at java.base/java.io.FileInputStream.open0(Native Method)
james1 | at java.base/java.io.FileInputStream.open(Unknown Source)
james1 | at java.base/java.io.FileInputStream.<init>(Unknown
Source)
james1 | at
org.apache.james.queue.activemq.FileSystemBlobStrategy.getInputStream(FileSystemBlobStrategy.java:101)
james1 | at
org.apache.activemq.blob.BlobDownloader.getInputStream(BlobDownloader.java:38)
james1 | at
org.apache.activemq.command.ActiveMQBlobMessage.getInputStream(ActiveMQBlobMessage.java:132)
james1 | at
org.apache.james.queue.activemq.MimeMessageBlobMessageSource.getInputStream(MimeMessageBlobMessageSource.java:45)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:307)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:296)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:292)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:276)
james1 | at
org.apache.james.server.core.MimeMessageWrapper.<init>(MimeMessageWrapper.java:158)
james1 | ... 19 common frames omitted
james1 | Wrapped by: javax.mail.MessagingException: IOException while
copying message
james1 | at
org.apache.james.server.core.MimeMessageWrapper.<init>(MimeMessageWrapper.java:165)
james1 | at
org.apache.james.server.core.MailImpl.setMessage(MailImpl.java:505)
james1 | at
com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73)
james1 | at java.base/java.util.Optional.ifPresent(Unknown Source)
james1 | at
org.apache.james.server.core.MailImpl$Builder.build(MailImpl.java:279)
james1 | at
org.apache.james.queue.jms.JMSCacheableMailQueue.createMail(JMSCacheableMailQueue.java:391)
james1 | at
org.apache.james.queue.activemq.ActiveMQCacheableMailQueue.createMailQueueItem(ActiveMQCacheableMailQueue.java:230)
james1 | at
org.apache.james.queue.jms.JMSCacheableMailQueue.deQueueOneItem(JMSCacheableMailQueue.java:249)
james1 | ... 12 common frames omitted
james1 | Wrapped by:
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue next
message
james1 | at
org.apache.james.queue.jms.JMSCacheableMailQueue.deQueueOneItem(JMSCacheableMailQueue.java:260)
james1 | at
reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
james1 | at
reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.resubscribe(FluxRepeatPredicate.java:120)
james1 | at
reactor.core.publisher.MonoRepeatPredicate.subscribeOrReturn(MonoRepeatPredicate.java:48)
james1 | at
reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:55)
james1 | at
reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
james1 | at
reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
james1 | at
reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
james1 | at java.base/java.util.concurrent.FutureTask.run(Unknown
Source)
james1 | at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
james1 | at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
james1 | at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
james1 | at java.base/java.lang.Thread.run(Unknown Source)
james1 | Wrapped by:
reactor.core.Exceptions$ErrorCallbackNotImplemented:
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue next
message;
james1 | nested exception is:
james1 | javax.mail.MessagingException: IOException while copying
message;
james1 | nested exception is:
james1 | java.io.FileNotFoundException:
/root/var/store/activemq/blob-transfer/10/ID_james1.local-33677-1677150190846-5_1_1_1_1037
(No such file or directory)
{code}
The issue is that blobs are handle via file so their deletion is not linked to
the JMS trasaction: the JMS transaction can be rolled back for a deleted blob,
thus leading to messages without blob.
Steps to reproduce:
- Send 5000 mails to james of 500 bytes each
{code:java}
smtp-source -A -C100 -r 1 -l 500 -m 5000 -s 20 -d -c -f [email protected] -M
other.com -t bob@localhost 172.24.0.4:25
{code}
- Once all mails are queued, restart james several time during processing
- Increasing the count of spooler workers helps.
As a fix I propose to discard emails not backed by a blob.
> ActiveMQ: restarting james can block mail queue
> -----------------------------------------------
>
> Key: JAMES-3891
> URL: https://issues.apache.org/jira/browse/JAMES-3891
> Project: James Server
> Issue Type: Improvement
> Components: Queue
> Affects Versions: master, 3.7.3
> Reporter: Benoit Tellier
> Priority: Major
> Fix For: 3.8.0, 3.7.4
>
>
> Given a mail queue currently containing emails and being consumed, When I
> restart James, Then after the restart the email consumptions sometime do not
> resume.
> The following log can be seen:
> {code:java}
> james1 | java.io.FileNotFoundException:
> /root/var/store/activemq/blob-transfer/10/ID_james1.local-33677-1677150190846-5_1_1_1_1037
> (No such file or directory)
> james1 | at java.base/java.io.FileInputStream.open0(Native
> Method)
> james1 | at java.base/java.io.FileInputStream.open(Unknown
> Source)
> james1 | at java.base/java.io.FileInputStream.<init>(Unknown
> Source)
> james1 | at
> org.apache.james.queue.activemq.FileSystemBlobStrategy.getInputStream(FileSystemBlobStrategy.java:101)
> james1 | at
> org.apache.activemq.blob.BlobDownloader.getInputStream(BlobDownloader.java:38)
> james1 | at
> org.apache.activemq.command.ActiveMQBlobMessage.getInputStream(ActiveMQBlobMessage.java:132)
> james1 | at
> org.apache.james.queue.activemq.MimeMessageBlobMessageSource.getInputStream(MimeMessageBlobMessageSource.java:45)
> james1 | at
> org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:307)
> james1 | at
> org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:296)
> james1 | at
> org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:292)
> james1 | at
> org.apache.james.server.core.MimeMessageWrapper.writeTo(MimeMessageWrapper.java:276)
> james1 | at
> org.apache.james.server.core.MimeMessageWrapper.<init>(MimeMessageWrapper.java:158)
> james1 | ... 19 common frames omitted
> james1 | Wrapped by: javax.mail.MessagingException: IOException
> while copying message
> james1 | at
> org.apache.james.server.core.MimeMessageWrapper.<init>(MimeMessageWrapper.java:165)
> james1 | at
> org.apache.james.server.core.MailImpl.setMessage(MailImpl.java:505)
> james1 | at
> com.github.fge.lambdas.consumers.ConsumerChainer.lambda$sneakyThrow$9(ConsumerChainer.java:73)
> james1 | at java.base/java.util.Optional.ifPresent(Unknown
> Source)
> james1 | at
> org.apache.james.server.core.MailImpl$Builder.build(MailImpl.java:279)
> james1 | at
> org.apache.james.queue.jms.JMSCacheableMailQueue.createMail(JMSCacheableMailQueue.java:391)
> james1 | at
> org.apache.james.queue.activemq.ActiveMQCacheableMailQueue.createMailQueueItem(ActiveMQCacheableMailQueue.java:230)
> james1 | at
> org.apache.james.queue.jms.JMSCacheableMailQueue.deQueueOneItem(JMSCacheableMailQueue.java:249)
> james1 | ... 12 common frames omitted
> james1 | Wrapped by:
> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue
> next message
> james1 | at
> org.apache.james.queue.jms.JMSCacheableMailQueue.deQueueOneItem(JMSCacheableMailQueue.java:260)
> james1 | at
> reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:44)
> james1 | at
> reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.resubscribe(FluxRepeatPredicate.java:120)
> james1 | at
> reactor.core.publisher.MonoRepeatPredicate.subscribeOrReturn(MonoRepeatPredicate.java:48)
> james1 | at
> reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:55)
> james1 | at
> reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
> james1 | at
> reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
> james1 | at
> reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
> james1 | at
> java.base/java.util.concurrent.FutureTask.run(Unknown Source)
> james1 | at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)
> james1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> james1 | at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> james1 | at java.base/java.lang.Thread.run(Unknown Source)
> james1 | Wrapped by:
> reactor.core.Exceptions$ErrorCallbackNotImplemented:
> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to dequeue
> next message;
> james1 | nested exception is:
> james1 | javax.mail.MessagingException: IOException while
> copying message;
> james1 | nested exception is:
> james1 | java.io.FileNotFoundException:
> /root/var/store/activemq/blob-transfer/10/ID_james1.local-33677-1677150190846-5_1_1_1_1037
> (No such file or directory)
> {code}
> No processing takes place and remaining emails are stuck into the mail queue.
> The issue is that blobs are handle via file so their deletion is not linked
> to the JMS trasaction: the JMS transaction can be rolled back for a deleted
> blob, thus leading to messages without blob.
> Steps to reproduce:
> - Send 5000 mails to james of 500 bytes each
> {code:java}
> smtp-source -A -C100 -r 1 -l 500 -m 5000 -s 20 -d -c -f [email protected] -M
> other.com -t bob@localhost 172.24.0.4:25
> {code}
> - Once all mails are queued, restart james several time during processing
> - Increasing the count of spooler workers helps.
> As a fix I propose to discard emails not backed by a blob.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
