Thanks to you both for your suggestions and questions.
Answers to questions
Regarding VM parameters and GC load, I'm confident I have this under
control. I have a lot of experience with GC tuning. I've been monitoring
this specific behaviour for a couple of months now in preparation for moving
WildFly into production. Clebert - you may recall I'm the one who spotted
the memory leak in ARTEMIS-320 by observing changed GC behaviour and went to
the effort to identify it down to the exact line of code. So you're right
to enquire about this but I know this is not a problem with GC settings.
Regarding my testing, I do burn-in for all tests and the JBoss versus
WildFly comparisons are on identical hardware using the same JDK version, GC
settings and load pattern (and as noted earlier are really only exercising
the JMS portions of the container). I'm not trying to argue that JBoss
messaging is better than Artemis but I do believe there is currently an
issue of some sort that is worth investigating.
Regarding my specific use case and the possibility of making a small
reproducer I can provide that if you confirm you still need it after reading
the next section.
Progress
As promised I ran some tests again whilst isolating the JMS and EJB clients.
This confirmed for certain that the issue is in the JMS/Netty not the
EJB/XNIO.
So the high number of PhantomReferences are being allocated in this stack:
Thread [Thread-1
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@788d198e-832298282)]
(Suspended (breakpoint at line 80 in PhantomReference))
Cleaner(PhantomReference<T>).<init>(T, ReferenceQueue<? super T>)
line: 80
Cleaner.<init>(Object, Runnable) line: 115
Cleaner.create(Object, Runnable) line: 133
DirectByteBuffer.<init>(int) line: 139
ByteBuffer.allocateDirect(int) line: 311
UnpooledUnsafeDirectByteBuf.allocateDirect(int) line: 108
UnpooledUnsafeDirectByteBuf.<init>(ByteBufAllocator, int, int) line:
69
UnpooledByteBufAllocator.newDirectBuffer(int, int) line: 50
UnpooledByteBufAllocator(AbstractByteBufAllocator).directBuffer(int,
int) line: 155
UnpooledByteBufAllocator(AbstractByteBufAllocator).directBuffer(int)
line: 146
NettyServerConnection.createTransportBuffer(int) line: 38
RemotingConnectionImpl(AbstractRemotingConnection).createTransportBuffer(int)
line: 162
SessionReceiveMessage.encode(RemotingConnection) line: 59
ChannelImpl.send(Packet, boolean, boolean) line: 225
ChannelImpl.sendBatched(Packet) line: 205
CoreSessionCallback.sendMessage(ServerMessage, ServerConsumer, int)
line: 84
ServerConsumerImpl.deliverStandardMessage(MessageReference,
ServerMessage) line: 883
ServerConsumerImpl.proceedDeliver(MessageReference) line: 366
QueueImpl.proceedDeliver(Consumer, MessageReference) line: 2358
QueueImpl.deliver() line: 1873
QueueImpl.access$1400(QueueImpl) line: 97
QueueImpl$DeliverRunner.run() line: 2581
OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run() line: 100
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1142
ThreadPoolExecutor$Worker.run() line: 617
Thread.run() line: 745
Every java.nio.DirectByteBuffer that is allocated is creating a
PhantomReference.
Clebert - In your earlier responses you refer to "netty for instance would
still be bootstrapping the reusable pool area" and "This could be Netty
bootstrapping the PooledByteBuffer arena", however the stack above suggests
that these ByteBuffers are not being pooled. *Note the
UnpooledByteBufAllocator*. This appears to be the default case for Netty
4.0 (http://netty.io/wiki/new-and-noteworthy-in-4.0.html#wiki-h2-3) but is
changed to default to pooled in 4.1
(http://netty.io/wiki/new-and-noteworthy-in-4.1.html#wiki-h3-7). So I think
this is our underlying problem. DirectByteBuffers are only recommended for
long-lived scenarios (per java.nio.ByteBuffer JavaDocs) which is not the
case if they are being used for only one message then discarded. It looks
like the NettyServerConnection should either be using HeapByteBuffers or we
should upgrade to Netty 4.1 so that the DirectByteBuffers get pooled and
reused.
Is there a way to set the Netty ALLOCATOR ChannelOption to
PooledByteBufAllocator via the command line so that I could test it with the
existing 4.0?
--
View this message in context:
http://activemq.2283324.n4.nabble.com/Artemis-use-of-PhantomReference-impacting-GC-performance-tp4706961p4707054.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
