On Mon, Nov 6, 2017 at 6:33 AM, Daniel Jeliński <djelins...@gmail.com> wrote:
> For others that run into similar issue, it turned out that the > OutOfMemoryError was thrown (and subsequently hidden) on the client side. > The error was caused by excessive direct memory usage in Java NIO's > bytebuffer caching (described here: > http://www.evanjones.ca/java-bytebuffer-leak.html), and setting > -Djdk.nio.maxCachedBufferSize=262144 > allowed the application to complete. > > Suggestions for how to expose the client-side OOME Daniel? We should add note to the thrown exception about "-Djdk.nio.maxCachedBufferSize" (and make sure the exception makes it out!) Thanks for updating the list, S > Yet another proof that correct handling of OOME is hard. > Thanks, > Daniel > > 2017-10-11 11:33 GMT+02:00 Daniel Jeliński <djelins...@gmail.com>: > > > Thanks for the hints. I'll see if we can explicitly set > > MaxDirectMemorySize to a safe number. > > Thanks, > > Daniel > > > > 2017-10-10 21:10 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>: > > > >> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/ > >> classes/sun/misc/VM.java#l184 > >> > >> // The initial value of this field is arbitrary; during JRE > >> initialization > >> // it will be reset to the value specified on the command line, if > >> any, > >> // otherwise to Runtime.getRuntime().maxMemory(). > >> > >> which goes all the way down to memory/heap.cpp to whatever was left to > the > >> reserved memory depending on the flags and the platform used as Vladimir > >> says. > >> > >> Also, depending on which distribution and features are used there are > >> specific guidelines about setting that parameter so mileage might vary. > >> > >> thanks, > >> esteban. > >> > >> > >> > >> -- > >> Cloudera, Inc. > >> > >> > >> On Tue, Oct 10, 2017 at 1:35 PM, Vladimir Rodionov < > >> vladrodio...@gmail.com> > >> wrote: > >> > >> > >> The default value is zero, which means the maximum direct memory is > >> > unbounded. > >> > > >> > That is not correct. If you do not specify MaxDirectMemorySize, > default > >> is > >> > platform specific > >> > > >> > The link above is for JRockit JVM I presume? > >> > > >> > On Tue, Oct 10, 2017 at 11:19 AM, Esteban Gutierrez < > >> este...@cloudera.com> > >> > wrote: > >> > > >> > > I don't think is truly unbounded, IIRC it s limited to the maximum > >> > > allocated heap. > >> > > > >> > > thanks, > >> > > esteban. > >> > > > >> > > -- > >> > > Cloudera, Inc. > >> > > > >> > > > >> > > On Tue, Oct 10, 2017 at 1:11 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > >> > > > >> > > > From https://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx. > >> htm : > >> > > > > >> > > > java -XX:MaxDirectMemorySize=2g myApp > >> > > > > >> > > > Default Value > >> > > > > >> > > > The default value is zero, which means the maximum direct memory > is > >> > > > unbounded. > >> > > > > >> > > > On Tue, Oct 10, 2017 at 11:04 AM, Vladimir Rodionov < > >> > > > vladrodio...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > >> XXMaxDirectMemorySize is set to the default 0, which means > >> > unlimited > >> > > > as > >> > > > > far > >> > > > > >> as I can tell. > >> > > > > > >> > > > > Not sure if this is true. The only conforming that link I found > >> was > >> > for > >> > > > > JRockit JVM. > >> > > > > > >> > > > > On Mon, Oct 9, 2017 at 11:29 PM, Daniel Jeliński < > >> > djelins...@gmail.com > >> > > > > >> > > > > wrote: > >> > > > > > >> > > > > > Vladimir, > >> > > > > > XXMaxDirectMemorySize is set to the default 0, which means > >> > unlimited > >> > > as > >> > > > > far > >> > > > > > as I can tell. > >> > > > > > Thanks, > >> > > > > > Daniel > >> > > > > > > >> > > > > > 2017-10-09 19:30 GMT+02:00 Vladimir Rodionov < > >> > vladrodio...@gmail.com > >> > > >: > >> > > > > > > >> > > > > > > Have you try to increase direct memory size for server > >> process? > >> > > > > > > -XXMaxDirectMemorySize=? > >> > > > > > > > >> > > > > > > On Mon, Oct 9, 2017 at 2:12 AM, Daniel Jeliński < > >> > > > djelins...@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hello, > >> > > > > > > > I'm running an application doing a lot of Puts (size > >> anywhere > >> > > > > between 0 > >> > > > > > > and > >> > > > > > > > 10MB, one cell at a time); occasionally I'm getting an > error > >> > like > >> > > > the > >> > > > > > > > below: > >> > > > > > > > 2017-10-09 04:29:29,811 WARN [AsyncProcess] - #13368, > >> > > > > > > > table=researchplatform:repo_stripe, attempt=1/1 > >> failed=1ops, > >> > > last > >> > > > > > > > exception: java.io.IOException: com.google.protobuf. > >> > > > > ServiceException: > >> > > > > > > > java.lang.OutOfMemoryError: Direct buffer memory on > >> > > > > > > > c169dzv.int.westgroup.com,60020,1506476748534, tracking > >> > started > >> > > > Mon > >> > > > > > Oct > >> > > > > > > 09 > >> > > > > > > > 04:29:29 EDT 2017; not retrying 1 - final failure > >> > > > > > > > > >> > > > > > > > After that the connection to RegionServer becomes > unusable. > >> > Every > >> > > > > > > > subsequent attempt to execute Put on that connection > >> results in > >> > > > > > > > CallTimeoutException. I only found the OutOfMemory by > >> reducing > >> > > the > >> > > > > > number > >> > > > > > > > of tries to 1. > >> > > > > > > > > >> > > > > > > > The host running HBase appears to have at least a few GB > of > >> > free > >> > > > > memory > >> > > > > > > > available. Server logs do not mention anything about this > >> > error. > >> > > > > > Cluster > >> > > > > > > is > >> > > > > > > > running HBase 1.2.0-cdh5.10.2. > >> > > > > > > > > >> > > > > > > > Is this a known problem? Are there workarounds available? > >> > > > > > > > Thanks, > >> > > > > > > > Daniel > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
