Seems like it might be related to https://bugs.eclipse.org/bugs/show_bug.cgi?id=424667
-- Joakim Erdfelt <[email protected]> webtide.com <http://www.webtide.com/> - intalio.com/jetty Expert advice, services and support from from the Jetty & CometD experts eclipse.org/jetty - cometd.org On Fri, Dec 27, 2013 at 1:23 AM, Jay C <[email protected]> wrote: > Hello everyone, > > I am new here. I recently started using Jetty 9.1 to serve a webapp making > intensive use of websockets with a certain success. Unfortunately the app > runs out of memory eventually. > > I have read Emmeran's recent post > < > http://jetty.4.x6.nabble.com/jetty-users-DirectByteBuffer-native-memory-problems-OOM-with-Jetty-9-0-9-1-tp4961615.html > > > about how "big writes" might lead to OOM issues. > > In my situation it is rather the opposite as my app makes many "small > writes". Typically, the webapp receives a stream of messages from a > backend > app via websocket and broadcasts them to the browsers via websocket as > well. > The browsers never send any data on the websocket. > > This is what happens after ~4 hours, there are 7M instances of > DirectByteBuffers: > > -Environment: > Jetty v9.1.0.v20131115 > JDK 1.7.0_45 > Ubuntu 12.10 x64 > > -Memory profile: > java.nio.DirectByteBuffer 465371136 > 465,371,136 B (55.2%) 7,271,424 (38.9%) > sun.misc.Cleaner 145428440 > 145,428,440 B (17.2%) 3,635,711 (19.5%) > java.nio.DirectByteBuffer$Deallocator 116342688 > 116,342,688 B (13.8%) 3,635,709 (19.5%) > java.util.concurrent.ConcurrentLinkedQueue$Node 87265560 87,265,560 B > (10.3%) 3,636,065 (19.5%) > > Caused by: java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > at > org.eclipse.jetty.util.BufferUtil.allocateDirect(BufferUtil.java:122) > at > > org.eclipse.jetty.io.MappedByteBufferPool.acquire(MappedByteBufferPool.java:59) > at > > org.eclipse.jetty.websocket.common.extensions.compress.DeflateFrameExtension.outgoingFrame(DeflateFrameExtension.java:173) > at > > org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:235) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendFrame(WebSocketRemoteEndpoint.java:168) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendAsyncFrame(WebSocketRemoteEndpoint.java:107) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendStringByFuture(WebSocketRemoteEndpoint.java:354) > > Switching between blocking and non-blocking writes won't change the > outcome. > > Each message is ~70 bytes before deflate and ~10 bytes after deflate. > Typically there are ~200 msg/s. > > > I observed that for each message: > > - AbstractWebSocketConnection acquires a buffer from ArrayByteBufferPool to > read the incoming message > > qtp1477190004-58@6990, prio=5, in group 'main', status: 'RUNNING' > at > > org.eclipse.jetty.io.*ArrayByteBufferPool.acquire*(ArrayByteBufferPool.java:64) > at > org.eclipse.jetty.websocket.common.Parser.parsePayload(Parser.java:563) > at > org.eclipse.jetty.websocket.common.Parser.parseFrame(Parser.java:525) > at > org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:240) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.read(AbstractWebSocketConnection.java:596) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:503) > at > org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) > at java.lang.Thread.run(Thread.java:744) > > - DeflateFrameExtension acquires and allocates a new direct buffer from > MappedByteBufferPool to write the outgoing message > > qtp1477190004-58@6990, prio=5, in group 'main', status: 'RUNNING' > at > org.eclipse.jetty.util.BufferUtil.allocateDirect(BufferUtil.java:122) > at > > org.eclipse.jetty.io.*MappedByteBufferPool.acquire*(MappedByteBufferPool.java:59) > at > > org.eclipse.jetty.websocket.common.extensions.compress.DeflateFrameExtension.outgoingFrame(DeflateFrameExtension.java:173) > at > > org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:235) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendFrame(WebSocketRemoteEndpoint.java:168) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:88) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:331) > /* handle incoming, crunch, send */ > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown > Source:-1) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > > org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71) > at > > org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72) > at > > org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryMessage(JettyAnnotatedEventDriver.java:107) > at > > org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.messageComplete(SimpleBinaryMessage.java:68) > at > > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:62) > at > > org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryFrame(JettyAnnotatedEventDriver.java:99) > at > > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:154) > at > > org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:300) > at > > org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202) > at > org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:206) > at > org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:243) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.read(AbstractWebSocketConnection.java:596) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:503) > at > org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) > at java.lang.Thread.run(Thread.java:744) > > - AbstractWebSocketConnection releases a buffer from ArrayByteBufferPool > after.. a successful write? > > qtp1477190004-58@6990, prio=5, in group 'main', status: 'RUNNING' > at > > org.eclipse.jetty.io.*ArrayByteBufferPool.release*(ArrayByteBufferPool.java:78) > at > > org.eclipse.jetty.websocket.common.io.WriteBytesProvider.releasePayloadBuffer(WriteBytesProvider.java:360) > at > > org.eclipse.jetty.websocket.common.io.WriteBytesProvider$FrameEntry.freeBuffers(WriteBytesProvider.java:105) > at > > org.eclipse.jetty.websocket.common.io.WriteBytesProvider$FrameEntry.notifySucceeded(WriteBytesProvider.java:83) > at > > org.eclipse.jetty.websocket.common.io.WriteBytesProvider.succeeded(WriteBytesProvider.java:407) > at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:357) > at > org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:125) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.write(AbstractWebSocketConnection.java:688) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.flush(AbstractWebSocketConnection.java:359) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:569) > at > > org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:168) > at > > org.eclipse.jetty.websocket.common.extensions.compress.DeflateFrameExtension.outgoingFrame(DeflateFrameExtension.java:213) > at > > org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:235) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendFrame(WebSocketRemoteEndpoint.java:168) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.blockingWrite(WebSocketRemoteEndpoint.java:88) > at > > org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendString(WebSocketRemoteEndpoint.java:331) > /* handle incoming, crunch, send */ > at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown > Source:-1) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > > org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:71) > at > > org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCallableMethod.java:72) > at > > org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryMessage(JettyAnnotatedEventDriver.java:107) > at > > org.eclipse.jetty.websocket.common.message.SimpleBinaryMessage.messageComplete(SimpleBinaryMessage.java:68) > at > > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:62) > at > > org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onBinaryFrame(JettyAnnotatedEventDriver.java:99) > at > > org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:154) > at > > org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:300) > at > > org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202) > at > org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:206) > at > org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:243) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.read(AbstractWebSocketConnection.java:596) > at > > org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:503) > at > org.eclipse.jetty.io.AbstractConnection$1.run(AbstractConnection.java:505) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) > at java.lang.Thread.run(Thread.java:744) > > > At this point I am starting to wonder if that is the intended behaviour ? > > - the MappedByteBufferPool buffers are always empty > > - MappedByteBufferPool.release() is never called, but > MappedByteBufferPool.acquire() is called for each > WebSocketRemoteEndPoint.sendString() > > - both Mapped and Array ByteBufferPool implementations are being used > throughout different parts of Jetty > > > Any idea regarding this issue anyone? > > Thanks. > > Jay > > > > -- > View this message in context: > http://jetty.4.x6.nabble.com/MappedByteBufferPool-not-pooling-tp4961821.html > Sent from the Jetty User mailing list archive at Nabble.com. > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users >
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
