Emmeran, on reflection, I think this is something we should deal with in jetty. I have opened: https://bugs.eclipse.org/bugs/show_bug.cgi?id=422807
and am working on a fix that will break up writes into getBufferSize() blocks, so that large direct buffers are not create/cached. If you want to write a large direct buffer, you can always create it yourself and use the jetty APIs to do the write (that will be really efficient!). cheers On 28 November 2013 10:49, Jan Bartel <[email protected]> wrote: > Hi Emmeran, > > This is a known bug in the jdk that has been around for quite a while. > The jetty doc on it is here (see subsection "Direct ByteBuffers"): > > http://www.eclipse.org/jetty/documentation/current/preventing-memory-leaks.html#jvm-bugs > > Sadly, there is still no fix in the jvm for it. If you've got a good > workaround, I'd go with that. > > cheers > Jan > > On 26 November 2013 21:10, Emmeran Seehuber <[email protected]> wrote: > > Hi, > > > > I already found the cause for this problem: I do big writes. > > > > The user upload wedding photos (the software is a web2print wedding card > shop), and I echo them back to the user. To do so I read them fully into a > byte[] and then do one big > > > > request.getOutputStream().write(data); > > > > As the images vary in size from 3 - 20+ MB, so do the byte[]… > > > > Jetty internally just wraps the byte[] into a ByteBuffer and passes it > to NIO. There it finally lands in > > > > IOUtil.write() > > http://www.docjar.com/html/api/sun/nio/ch/IOUtil.java.html > > > > If the ByteBuffer is a DirectBuffer, everything is fine, and the whole > thing is passed to the native write. But if it is not, it allocates a > temporary DirectByteBuffer of the full size of the ByteBuffer using > > > > Util.getTemporaryDirectBuffer > > http://www.docjar.com/html/api/sun/nio/ch/Util.java.html > > > > The problem is, this DirectByteBuffer are then after the write cached in > a thread local pool of 8 such DirectByteBuffer. The pool keeps the biggest > 8 requested buffers. > > > > If you now have about 200+ live threads and any of them can do a 8, 10, > … MB write, then you end up using much direct memory. Over time this pool > just grows and grows …. > > > > After trying to avoid reading the image data into a byte[] as much as > possible and if i have to read the image fully into a byte[] just doing > chunked writes of the byte buffer, the problem is gone. I.e. i do now a > > > > ServletOutputStream outputStream = res.getOutputStream(); > > int i = 0; > > final int MAX_BYTES_TO_SEND = 64 * 1024; > > while (i < data.length) { > > final int maxSendSize = > Math.min(MAX_BYTES_TO_SEND, data.length - i); > > outputStream.write(data, i, maxSendSize); > > i += maxSendSize; > > } > > > > and everything is fine. > > > > Of course, this was no problem with Jetty 7, because there I did not use > the NIO connector there. > > > > I am not sure if you can consider this a bug of Jetty, or if this is > rather a bug of the JDK. At least it is very unexpected. > > > > Maybe Jetty should workaround this buggy behavior and just always use > its internal DirectByteBuffers and do chunked writes with DirectByteBuffers > itself? > > > > I´m not sure if the JDK behavior gives any speed advantages if you cache > DirectByteBuffers bigger then a certain size (e.g. 64KB) - or just simple > trashes the CPU cache instead ... > > > > Should I still open a bug for this? > > > > Thanks. > > > > cu, > > Emmy > > > > (BTW: The OS is Linux) > > > > Am 25.11.2013 um 02:37 schrieb Jan Bartel <[email protected]>: > > > >> And when you open the bug, please make sure to state which operating > >> system and version you are using. > >> > >> thanks > >> Jan > >> > >> On 24 November 2013 09:55, Simone Bordet <[email protected]> wrote: > >>> Hi, > >>> > >>> On Thu, Nov 21, 2013 at 2:47 PM, Emmeran Seehuber <[email protected]> > wrote: > >>>> Hello everybody, > >>>> > >>>> I´m using jetty as standalone embedded web server behind a nginx and > it is > >>>> working great so far. Recently I upgraded from 7.6.4 to 9.0 and two > days ago > >>>> to 9.1.0.v20131115. > >>>> > >>>> Since my upgrade to 9.0 i experience native memory leaks. Also with > 9.1 i > >>>> still have the same leaks. Native memory leaks mean, that the process > memory > >>>> size grows very large and then I get OOM Exceptions for > DirectByteBuffers. > >>>> But to get there the server process has to run for about two weeks. > >>>> > >>>> Since the process has a 7 GB heap, the MaxDirectMemory Limit also > seems to > >>>> be 7 GB. > >>>> > >>>> First i thought that the many threads started/stopped in the thread > pool are > >>>> the problem. I specified a idle time of -1, and still many threads are > >>>> restarted. Also some answers on the web about DirectByteBuffer OOM > >>>> Exceptions pointed into this direction. But i don’t think thats the > problem. > >>>> > >>>> I think the problem may be in ArrayByteBufferPool. It allocates an > unbound > >>>> number of ByteBuffers and caches them in a unbound > ConcurrentLinkedQueue. So > >>>> at some peak times many buffers are used and never freed. On the > other side > >>>> this doesn’t really fit with the allocated memory. At the moment the > server > >>>> has this direct memory allocations (according to the MBeans info): > >>>> > >>>> java.nio > >>>> BufferPool > >>>> name=direct > >>>> - Count 384 > >>>> - MemoryUsed 2728821147 > >>>> - Name direct > >>>> - ObjectName java.nio:type=BufferPool,name=direct > >>>> - TotalCapacity 2728821147 > >>>> > >>>> That would mean an medium buffer size of about 7 MB. But the > >>>> ArrayByteBufferPool does, as far as i understand, only allocate a max > of 64 > >>>> kb sized buffers by default ?! > >>>> > >>>> Current threads are: Count = 230, Maximum = 233, Started overall = 599 > >>>> > >>>> This numbers are after the process is running for about 32 hours. > I´ve got > >>>> about 320 request/minute at peak times. All this numbers are > according to > >>>> the embedded JavaMelody monitoring. > >>>> > >>>> The JVM arguments are: > >>>> -server -XX:+UseCompressedOops -XX:MaxPermSize=512m -Xms7000M > -Xmx7000M > >>>> -XX:+UseParallelOldGC -XX:+DoEscapeAnalysis -XX:+OptimizeStringConcat > >>>> JDK 1.7.0_45 > >>>> > >>>> The thread pool is configured this way: > >>>> > >>>> tp = new QueuedThreadPool(); > >>>> tp.setMaxThreads(256); > >>>> tp.setMinThreads(5); > >>>> tp.setIdleTimeout(-1); > >>>> tp.setName("Http Server Thread Pool"); > >>>> tp.setDaemon(true); > >>>> > >>>> I had no such problems with jetty 7.x. Regularly calling System.gc() > every > >>>> hour does not help. Also the i´ve got no other GC problems. > >>>> > >>>> Any ideas what could cause this problems? > >>>> Why are threads restarted in the pool even when i specify a idle > timeout of > >>>> -1? I also regulary schedule some own runnables on the thread pool - > but > >>>> they should not cause this problems, should they? They do in 99.99% > of all > >>>> cases not throw any exceptions. > >>> > >>> Seems like a bug. > >>> I would keep the ThreadPool at default configuration for now, to have > >>> less variables in the system. > >>> > >>> Would you be able to replace usage of ArrayByteBufferPool with > >>> MappedByteBufferPool and see if the problem persist ? > >>> > >>> Also, please file an issue about this, it really looks from JMX that > >>> there is a problem. > >>> > >>> Thanks ! > >>> > >>> -- > >>> Simone Bordet > >>> ---- > >>> http://cometd.org > >>> http://webtide.com > >>> http://intalio.com > >>> Developer advice, training, services and support > >>> from the Jetty & CometD experts. > >>> Intalio, the modern way to build business applications. > >>> _______________________________________________ > >>> jetty-users mailing list > >>> [email protected] > >>> https://dev.eclipse.org/mailman/listinfo/jetty-users > >> > >> > >> > >> -- > >> Jan Bartel <[email protected]> > >> www.webtide.com > >> 'Expert Jetty/CometD developer,production,operations advice' > >> _______________________________________________ > >> jetty-users mailing list > >> [email protected] > >> https://dev.eclipse.org/mailman/listinfo/jetty-users > >> > > > > Mit freundlichen Grüßen aus Augsburg, > > > > Emmeran Seehuber > > Dipl. Inf. (FH) > > Schrannenstraße 8 > > 86150 Augsburg > > USt-IdNr.: DE266070804 > > > > _______________________________________________ > > jetty-users mailing list > > [email protected] > > https://dev.eclipse.org/mailman/listinfo/jetty-users > > > > -- > Jan Bartel <[email protected]> > www.webtide.com > 'Expert Jetty/CometD developer,production,operations advice' > _______________________________________________ > jetty-users mailing list > [email protected] > https://dev.eclipse.org/mailman/listinfo/jetty-users > -- Greg Wilkins <[email protected]> http://www.webtide.com Developer advice and support from the Jetty & CometD experts. Intalio, the modern way to build business applications.
_______________________________________________ jetty-users mailing list [email protected] https://dev.eclipse.org/mailman/listinfo/jetty-users
