Hi Emmeran, This is a known bug in the jdk that has been around for quite a while. The jetty doc on it is here (see subsection "Direct ByteBuffers"): http://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html#jvm-bugs
Sadly, there is still no fix in the jvm for it. If you've got a good workaround, I'd go with that. cheers Jan On 26 November 2013 21:10, Emmeran Seehuber <[email protected]> wrote: > Hi, > > I already found the cause for this problem: I do big writes. > > The user upload wedding photos (the software is a web2print wedding card > shop), and I echo them back to the user. To do so I read them fully into a > byte[] and then do one big > > request.getOutputStream().write(data); > > As the images vary in size from 3 - 20+ MB, so do the byte[]… > > Jetty internally just wraps the byte[] into a ByteBuffer and passes it to > NIO. There it finally lands in > > IOUtil.write() > http://www.docjar.com/html/api/sun/nio/ch/IOUtil.java.html > > If the ByteBuffer is a DirectBuffer, everything is fine, and the whole thing > is passed to the native write. But if it is not, it allocates a temporary > DirectByteBuffer of the full size of the ByteBuffer using > > Util.getTemporaryDirectBuffer > http://www.docjar.com/html/api/sun/nio/ch/Util.java.html > > The problem is, this DirectByteBuffer are then after the write cached in a > thread local pool of 8 such DirectByteBuffer. The pool keeps the biggest 8 > requested buffers. > > If you now have about 200+ live threads and any of them can do a 8, 10, … MB > write, then you end up using much direct memory. Over time this pool just > grows and grows …. > > After trying to avoid reading the image data into a byte[] as much as > possible and if i have to read the image fully into a byte[] just doing > chunked writes of the byte buffer, the problem is gone. I.e. i do now a > > ServletOutputStream outputStream = res.getOutputStream(); > int i = 0; > final int MAX_BYTES_TO_SEND = 64 * 1024; > while (i < data.length) { > final int maxSendSize = Math.min(MAX_BYTES_TO_SEND, > data.length - i); > outputStream.write(data, i, maxSendSize); > i += maxSendSize; > } > > and everything is fine. > > Of course, this was no problem with Jetty 7, because there I did not use the > NIO connector there. > > I am not sure if you can consider this a bug of Jetty, or if this is rather a > bug of the JDK. At least it is very unexpected. > > Maybe Jetty should workaround this buggy behavior and just always use its > internal DirectByteBuffers and do chunked writes with DirectByteBuffers > itself? > > I´m not sure if the JDK behavior gives any speed advantages if you cache > DirectByteBuffers bigger then a certain size (e.g. 64KB) - or just simple > trashes the CPU cache instead ... > > Should I still open a bug for this? > > Thanks. > > cu, > Emmy > > (BTW: The OS is Linux) > > Am 25.11.2013 um 02:37 schrieb Jan Bartel <[email protected]>: > >> And when you open the bug, please make sure to state which operating >> system and version you are using. >> >> thanks >> Jan >> >> On 24 November 2013 09:55, Simone Bordet <[email protected]> wrote: >>> Hi, >>> >>> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <[email protected]> >>> wrote: >>>> Hello everybody, >>>> >>>> I´m using jetty as standalone embedded web server behind a nginx and it is >>>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two days >>>> ago >>>> to 9.1.0.v20131115. >>>> >>>> Since my upgrade to 9.0 i experience native memory leaks. Also with 9.1 i >>>> still have the same leaks. Native memory leaks mean, that the process >>>> memory >>>> size grows very large and then I get OOM Exceptions for DirectByteBuffers. >>>> But to get there the server process has to run for about two weeks. >>>> >>>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also seems to >>>> be 7 GB. >>>> >>>> First i thought that the many threads started/stopped in the thread pool >>>> are >>>> the problem. I specified a idle time of -1, and still many threads are >>>> restarted. Also some answers on the web about DirectByteBuffer OOM >>>> Exceptions pointed into this direction. But i don’t think thats the >>>> problem. >>>> >>>> I think the problem may be in ArrayByteBufferPool. It allocates an unbound >>>> number of ByteBuffers and caches them in a unbound ConcurrentLinkedQueue. >>>> So >>>> at some peak times many buffers are used and never freed. On the other >>>> side >>>> this doesn’t really fit with the allocated memory. At the moment the server >>>> has this direct memory allocations (according to the MBeans info): >>>> >>>> java.nio >>>> BufferPool >>>> name=direct >>>> - Count 384 >>>> - MemoryUsed 2728821147 >>>> - Name direct >>>> - ObjectName java.nio:type=BufferPool,name=direct >>>> - TotalCapacity 2728821147 >>>> >>>> That would mean an medium buffer size of about 7 MB. But the >>>> ArrayByteBufferPool does, as far as i understand, only allocate a max of 64 >>>> kb sized buffers by default ?! >>>> >>>> Current threads are: Count = 230, Maximum = 233, Started overall = 599 >>>> >>>> This numbers are after the process is running for about 32 hours. I´ve got >>>> about 320 request/minute at peak times. All this numbers are according to >>>> the embedded JavaMelody monitoring. >>>> >>>> The JVM arguments are: >>>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M -Xmx7000M >>>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat >>>> JDK 1.7.0_45 >>>> >>>> The thread pool is configured this way: >>>> >>>> tp = new QueuedThreadPool(); >>>> tp.setMaxThreads(256); >>>> tp.setMinThreads(5); >>>> tp.setIdleTimeout(-1); >>>> tp.setName("Http Server Thread Pool"); >>>> tp.setDaemon(true); >>>> >>>> I had no such problems with jetty 7.x. Regularly calling System.gc() every >>>> hour does not help. Also the i´ve got no other GC problems. >>>> >>>> Any ideas what could cause this problems? >>>> Why are threads restarted in the pool even when i specify a idle timeout of >>>> -1? I also regulary schedule some own runnables on the thread pool - but >>>> they should not cause this problems, should they? They do in 99.99% of all >>>> cases not throw any exceptions. >>> >>> Seems like a bug. >>> I would keep the ThreadPool at default configuration for now, to have >>> less variables in the system. >>> >>> Would you be able to replace usage of ArrayByteBufferPool with >>> MappedByteBufferPool and see if the problem persist ? >>> >>> Also, please file an issue about this, it really looks from JMX that >>> there is a problem. >>> >>> Thanks ! >>> >>> -- >>> Simone Bordet >>> ---- >>> http://cometd.org >>> http://webtide.com >>> http://intalio.com >>> Developer advice, training, services and support >>> from the Jetty & CometD experts. >>> Intalio, the modern way to build business applications. >>> _______________________________________________ >>> jetty-users mailing list >>> [email protected] >>> https://dev.eclipse.org/mailman/listinfo/jetty-users >> >> >> >> -- >> Jan Bartel <[email protected]> >> www.webtide.com >> 'Expert Jetty/CometD developer,production,operations advice' >> _______________________________________________ >> jetty-users mailing list >> [email protected] >> https://dev.eclipse.org/mailman/listinfo/jetty-users >> > > Mit freundlichen Grüßen aus Augsburg, > > Emmeran Seehuber > Dipl. Inf. (FH) > Schrannenstraße 8 > 86150 Augsburg > USt-IdNr.: DE266070804 > > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users -- Jan Bartel <[email protected]> www.webtide.com 'Expert Jetty/CometD developer,production,operations advice' _______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
