Hello Wojtek,
I am taking a look into this, running a JPA james server localy, and
synchronizing emails with a remote server, I also notice a memory leak.
I opened an issue to track this:
https://issues.apache.org/jira/browse/JAMES-3911
I suspect https://github.com/apache/james-project/pull/1574 could be
related...
Best regards
On 02/06/2023 08:58, Wojtek wrote:
Hi,
I was attempting to transfer data from my old account to James and ran
into an issue: "OutOfMemoryError: Java heap space". Upon closer
inspection, James was using 4,7g virt / 2,3g res memory (system usage
of the generated docker image was 7,6g with 0,1g free reported). This
is a fairly simple setup of james-jpa backed by MySQL.
Has anyone ran into similar problem? imap sync seemed to process
messages one-by-one, so the load wasn't great. I'm currently
collecting heapdump.
Error stacktraces entry seems to point to MySQL/openJPA - one
assumption would be that James itself can process incomming messages
faster than the repository can process/store them? Isn't there a
feature that would slow down processing incomming messages in thise case?
21:44:10.161 [WARN ] r.c.Exceptions - throwIfFatal detected a jvm
fatal exception, which is thrown and logged below:
java.lang.OutOfMemoryError: Java heap space
at java.base/java.math.BigInteger.shiftRightImpl(Unknown Source)
at java.base/java.math.BigInteger.shiftRight(Unknown Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValuePositive(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValue(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.setValue(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.<init>(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial$ImmutableElement.<init>(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown
Source)
at
jdk.crypto.ec/sun.security.ec.XECOperations.encodedPointMultiply(Unknown
Source)
at
jdk.crypto.ec/sun.security.ec.XDHKeyAgreement.engineDoPhase(Unknown
Source)
at java.base/javax.crypto.KeyAgreement.doPhase(Unknown Source)
at
java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(Unknown Source)
at
java.base/sun.security.ssl.KAKeyDerivation.deriveKey(Unknown Source)
at
java.base/sun.security.ssl.ServerHello$T13ServerHelloConsumer.consume(Unknown
Source)
at
java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(Unknown
Source)
at
java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(Unknown
Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown
Source)
at
java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at
java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at
java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown
Source)
at
java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown
Source)
at
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at
com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:347)
at
com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:194)
at
com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:101)
at
com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:308)
at
com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:204)
at
com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1369)
21:44:14.037 [ERROR] r.c.s.Schedulers - Scheduler worker in group main
failed with an uncaught exception
java.lang.OutOfMemoryError: Java heap space
at java.base/java.math.BigInteger.shiftRightImpl(Unknown Source)
at java.base/java.math.BigInteger.shiftRight(Unknown Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValuePositive(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.setLimbsValue(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.setValue(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial$Element.<init>(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial$ImmutableElement.<init>(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown
Source)
at
java.base/sun.security.util.math.intpoly.IntegerPolynomial.getElement(Unknown
Source)
at
jdk.crypto.ec/sun.security.ec.XECOperations.encodedPointMultiply(Unknown
Source)
at
jdk.crypto.ec/sun.security.ec.XDHKeyAgreement.engineDoPhase(Unknown
Source)
at java.base/javax.crypto.KeyAgreement.doPhase(Unknown Source)
at
java.base/sun.security.ssl.KAKeyDerivation.t13DeriveKey(Unknown Source)
at
java.base/sun.security.ssl.KAKeyDerivation.deriveKey(Unknown Source)
at
java.base/sun.security.ssl.ServerHello$T13ServerHelloConsumer.consume(Unknown
Source)
at
java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(Unknown
Source)
at
java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(Unknown
Source)
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown
Source)
at
java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at
java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
at
java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown
Source)
at
java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown
Source)
at
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at
java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
at
com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:347)
at
com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:194)
at
com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:101)
at
com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:308)
at
com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:204)
at
com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1369)
21:47:57.114 [ERROR] o.a.j.i.p.AbstractMailboxProcessor - Unexpected
error during IMAP processing
org.apache.openjpa.lib.jdbc.ReportingSQLException: Java heap space
{prepstmnt 2016315081 SELECT t0.MAILBOX_ID, t0.MAIL_UID,
t0.MAIL_IS_ANSWERED, t0.MAIL_BODY_START_OCTET,
t0.MAIL_CONTENT_OCTETS_COUNT, t0.MAIL_IS_DELETED, t0.MAIL_IS_DRAFT,
t0.MAIL_IS_FLAGGED, t0.MAIL_DATE, t1.MAILBOX_ID,
t1.MAILBOX_HIGHEST_MODSEQ, t1.MAILBOX_LAST_UID, t1.MAILBOX_NAME,
t1.MAILBOX_NAMESPACE, t1.MAILBOX_UID_VALIDITY, t1.USER_NAME,
t0.MAIL_MIME_TYPE, t0.MAIL_MODSEQ, t0.MAIL_IS_RECENT, t0.MAIL_IS_SEEN,
t0.MAIL_MIME_SUBTYPE, t0.MAIL_TEXTUAL_LINE_COUNT FROM JAMES_MAIL t0
LEFT OUTER JOIN JAMES_MAILBOX t1 ON t0.MAILBOX_ID = t1.MAILBOX_ID
WHERE (t0.MAILBOX_ID = ?) ORDER BY t0.MAIL_UID ASC} [code=0, state=S1000]
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:219)
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:203)
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:58)
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeQuery(LoggingConnectionDecorator.java:1171)
at
org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:300)
at
org.apache.openjpa.jdbc.kernel.JDBCStoreManager$CancelPreparedStatement.executeQuery(JDBCStoreManager.java:1870)
at
org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:290)
at
org.apache.openjpa.jdbc.sql.SelectImpl.executeQuery(SelectImpl.java:531)
at
org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:456)
at
org.apache.openjpa.jdbc.sql.SelectImpl.execute(SelectImpl.java:423)
at
org.apache.openjpa.jdbc.sql.LogicalUnion$UnionSelect.execute(LogicalUnion.java:477)
at
org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:259)
at
org.apache.openjpa.jdbc.sql.LogicalUnion.execute(LogicalUnion.java:248)
at
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.open(SelectResultObjectProvider.java:95)
at
org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:36)
... 78 common frames omitted
Wrapped by: org.apache.openjpa.persistence.PersistenceException: Java
heap space {prepstmnt 2016315081 SELECT t0.MAILBOX_ID, t0.MAIL_UID,
t0.MAIL_IS_ANSWERED, t0.MAIL_BODY_START_OCTET,
t0.MAIL_CONTENT_OCTETS_COUNT,
t0.MAIL_IS_DELETED, t0.MAIL_IS_DRAFT, t0.MAIL_IS_FLAGGED,
t0.MAIL_DATE, t1.MAILBOX_ID, t1.MAILBOX_HIGHEST_MODSEQ,
t1.MAILBOX_LAST_UID, t1.MAILBOX_NAME, t1.MAILBOX_NAMESPACE,
t1.MAILBOX_UID_VALIDITY, t1.USER_NAME,
t0.MAIL_MIME_TYPE, t0.MAIL_MODSEQ, t0.MAIL_IS_RECENT, t0.MAIL_IS_SEEN,
t0.MAIL_MIME_SUBTYPE, t0.MAIL_TEXTUAL_LINE_COUNT FROM JAMES_MAIL t0
LEFT OUTER JOIN JAMES_MAILBOX t1 ON t0.MAILBOX_ID = t1.MAILBOX_ID
WHERE (t0
.MAILBOX_ID = ?) ORDER BY t0.MAIL_UID ASC} [code=0, state=S1000]
at
org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:5320)
at
org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:5280)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:134)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:115)
at
org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:67)
at
org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.handleCheckedException(SelectResultObjectProvider.java:162)
at
org.apache.openjpa.lib.rop.EagerResultList.<init>(EagerResultList.java:42)
at
org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1314)
at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1061)
at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:911)
at
org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:842)
at
org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:601)
at
org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:297)
at
org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:314)
at
org.apache.james.mailbox.jpa.mail.JPAMessageMapper.findMessagesInMailbox(JPAMessageMapper.java:414)
at
org.apache.james.mailbox.jpa.mail.JPAMessageMapper.findAsList(JPAMessageMapper.java:113)
at
org.apache.james.mailbox.jpa.mail.JPAMessageMapper.findInMailbox(JPAMessageMapper.java:100)
at
org.apache.james.mailbox.store.mail.MessageMapper.findInMailboxReactive(MessageMapper.java:88)
at
org.apache.james.mailbox.jpa.mail.JPAMessageMapper.listAllMessageUids(JPAMessageMapper.java:92)
at
org.apache.james.mailbox.store.StoreMessageManager.lambda$listAllMessageUids$23(StoreMessageManager.java:986)
at
org.apache.james.mailbox.store.transaction.TransactionalMapper.execute(TransactionalMapper.java:38)
... 64 common frames omitted
Wrapped by: org.apache.openjpa.persistence.RollbackException: Java
heap space {prepstmnt 2016315081 SELECT t0.MAILBOX_ID, t0.MAIL_UID,
t0.MAIL_IS_ANSWERED, t0.MAIL_BODY_START_OCTET,
t0.MAIL_CONTENT_OCTETS_COUNT, t0
.MAIL_IS_DELETED, t0.MAIL_IS_DRAFT, t0.MAIL_IS_FLAGGED, t0.MAIL_DATE,
t1.MAILBOX_ID, t1.MAILBOX_HIGHEST_MODSEQ, t1.MAILBOX_LAST_UID,
t1.MAILBOX_NAME, t1.MAILBOX_NAMESPACE, t1.MAILBOX_UID_VALIDITY,
t1.USER_NAME, t0.
MAIL_MIME_TYPE, t0.MAIL_MODSEQ, t0.MAIL_IS_RECENT, t0.MAIL_IS_SEEN,
t0.MAIL_MIME_SUBTYPE, t0.MAIL_TEXTUAL_LINE_COUNT FROM JAMES_MAIL t0
LEFT OUTER JOIN JAMES_MAILBOX t1 ON t0.MAILBOX_ID = t1.MAILBOX_ID
WHERE (t0.MA
ILBOX_ID = ?) ORDER BY t0.MAIL_UID ASC} [code=0, state=S1000]
at
org.apache.openjpa.persistence.EntityManagerImpl.commit(EntityManagerImpl.java:713)
at
org.apache.james.mailbox.jpa.JPATransactionalMapper.commit(JPATransactionalMapper.java:71)
... 65 common frames omitted
Wrapped by: org.apache.james.mailbox.exception.MailboxException:
Commit of transaction failed
at
org.apache.james.mailbox.jpa.JPATransactionalMapper.commit(JPATransactionalMapper.java:73)
at
org.apache.james.mailbox.store.transaction.TransactionalMapper.execute(TransactionalMapper.java:39)
at
org.apache.james.mailbox.store.StoreMessageManager.listAllMessageUids(StoreMessageManager.java:985)
at
org.apache.james.mailbox.store.StoreMessageManager.search(StoreMessageManager.java:835)
at
org.apache.james.mailbox.store.StoreMessageManager.search(StoreMessageManager.java:132)
at
org.apache.james.imap.processor.base.SelectedMailboxImpl.finishInit(SelectedMailboxImpl.java:167)
at
org.apache.james.imap.processor.AbstractSelectionProcessor.selectMailbox(AbstractSelectionProcessor.java:426)
at
org.apache.james.imap.processor.AbstractSelectionProcessor.lambda$selectMailbox$25(AbstractSelectionProcessor.java:399)
at
com.github.fge.lambdas.functions.FunctionChainer.doApply(FunctionChainer.java:20)
at
com.github.fge.lambdas.functions.ThrowingFunction.apply(ThrowingFunction.java:17)
... 56 common frames omitted
Wrapped by: com.github.fge.lambdas.ThrownByLambdaException:
org.apache.james.mailbox.exception.MailboxException: Commit of
transaction failed
at
com.github.fge.lambdas.functions.ThrowingFunction.apply(ThrowingFunction.java:21)
at
reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
at
reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
at
reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
at
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
at
reactor.core.publisher.MonoCallable$MonoCallableSubscription.request(MonoCallable.java:156)
at
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2341)
at
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74)
at
reactor.core.publisher.MonoCallable.subscribe(MonoCallable.java:48)
at
reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:109)
at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:102)
at
reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:240)
at
reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
at reactor.core.publisher.Mono.subscribe(Mono.java:4444)
at reactor.core.publisher.Mono.subscribeWith(Mono.java:4510)
at reactor.core.publisher.Mono.subscribe(Mono.java:4272)
at
org.apache.james.imapserver.netty.ImapChannelUpstreamHandler.channelRead(ImapChannelUpstreamHandler.java:367)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
at
io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
at
io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.base/java.lang.Thread.run(Unknown Source)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscr...@james.apache.org
For additional commands, e-mail: server-user-h...@james.apache.org
