Re: OutOfMemoryError: Direct buffer memory on PUT

We run into one similar case for the replication at the DR cluster. Turned out that I filed HBASE-19320 without knowing works done here. The way I detected the DM leak is using the metrix to find the direct memory usage and heap dump to analyze

Re: OutOfMemoryError: Direct buffer memory on PUT

On Wed, Nov 8, 2017 at 3:31 AM, Abhishek Singh Chouhan < abhishekchouhan...@gmail.com> wrote: > I faced the same issue and have been debugging this for some time now(the > logging is not very helpful as daniel mentions :)). > Looking deeper into this i realized that the side effects also are

Re: OutOfMemoryError: Direct buffer memory on PUT

I faced the same issue and have been debugging this for some time now(the logging is not very helpful as daniel mentions :)). Looking deeper into this i realized that the side effects also are large incorrect byte buffer allocations on the server side apart from call timeouts on the client side.

Re: OutOfMemoryError: Direct buffer memory on PUT

2017-11-07 18:22 GMT+01:00 Stack : > On Mon, Nov 6, 2017 at 6:33 AM, Daniel Jeliński > wrote: > > > For others that run into similar issue, it turned out that the > > OutOfMemoryError was thrown (and subsequently hidden) on the client side. > > The error

Re: OutOfMemoryError: Direct buffer memory on PUT

On Mon, Nov 6, 2017 at 6:33 AM, Daniel Jeliński wrote: > For others that run into similar issue, it turned out that the > OutOfMemoryError was thrown (and subsequently hidden) on the client side. > The error was caused by excessive direct memory usage in Java NIO's >

Re: OutOfMemoryError: Direct buffer memory on PUT

For others that run into similar issue, it turned out that the OutOfMemoryError was thrown (and subsequently hidden) on the client side. The error was caused by excessive direct memory usage in Java NIO's bytebuffer caching (described here: http://www.evanjones.ca/java-bytebuffer-leak.html), and

Re: OutOfMemoryError: Direct buffer memory on PUT

Thanks for the hints. I'll see if we can explicitly set MaxDirectMemorySize to a safe number. Thanks, Daniel 2017-10-10 21:10 GMT+02:00 Esteban Gutierrez : > http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/ > share/classes/sun/misc/VM.java#l184 > > // The initial

Re: OutOfMemoryError: Direct buffer memory on PUT

http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/sun/misc/VM.java#l184 // The initial value of this field is arbitrary; during JRE initialization // it will be reset to the value specified on the command line, if any, // otherwise to

Re: OutOfMemoryError: Direct buffer memory on PUT

>> The default value is zero, which means the maximum direct memory is unbounded. That is not correct. If you do not specify MaxDirectMemorySize, default is platform specific The link above is for JRockit JVM I presume? On Tue, Oct 10, 2017 at 11:19 AM, Esteban Gutierrez

Re: OutOfMemoryError: Direct buffer memory on PUT

I don't think is truly unbounded, IIRC it s limited to the maximum allocated heap. thanks, esteban. -- Cloudera, Inc. On Tue, Oct 10, 2017 at 1:11 PM, Ted Yu wrote: > From https://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm : > > java

Re: OutOfMemoryError: Direct buffer memory on PUT

>From https://docs.oracle.com/cd/E15289_01/doc.40/e15062/optionxx.htm : java -XX:MaxDirectMemorySize=2g myApp Default Value The default value is zero, which means the maximum direct memory is unbounded. On Tue, Oct 10, 2017 at 11:04 AM, Vladimir Rodionov wrote: > >>

Re: OutOfMemoryError: Direct buffer memory on PUT

>> XXMaxDirectMemorySize is set to the default 0, which means unlimited as far >> as I can tell. Not sure if this is true. The only conforming that link I found was for JRockit JVM. On Mon, Oct 9, 2017 at 11:29 PM, Daniel Jeliński wrote: > Vladimir, >

Re: OutOfMemoryError: Direct buffer memory on PUT

Vladimir, XXMaxDirectMemorySize is set to the default 0, which means unlimited as far as I can tell. Thanks, Daniel 2017-10-09 19:30 GMT+02:00 Vladimir Rodionov : > Have you try to increase direct memory size for server process? > -XXMaxDirectMemorySize=? > > On Mon, Oct

Re: OutOfMemoryError: Direct buffer memory on PUT

Ted, Apparently it does not; BoundedByteBufferPool does not contain any references to allocateDirect. Thanks, Daniel 2017-10-09 19:41 GMT+02:00 Ted Yu : > Daniel: > Does the version you use contain HBASE-13819 ? > > Cheers > > On Mon, Oct 9, 2017 at 2:12 AM, Daniel Jeliński

Re: OutOfMemoryError: Direct buffer memory on PUT

Sudhir, Not on the cluster in question. It's still configured to use HFile v2, and as far as I can tell, this is not a setting you could override on a table level. Thanks, Daniel 2017-10-10 0:16 GMT+02:00 sudhir patil : > If your cell size is greater than 100KB its

Re: OutOfMemoryError: Direct buffer memory on PUT

If your cell size is greater than 100KB its recommended to use MOB's https://www.cloudera.com/documentation/enterprise/5-4-x/topics/admin_hbase_mob.html, can you give it a try? On Tue, Oct 10, 2017 at 1:41 AM, Ted Yu wrote: > Daniel: > Does the version you use contain

Re: OutOfMemoryError: Direct buffer memory on PUT

Daniel: Does the version you use contain HBASE-13819 ? Cheers On Mon, Oct 9, 2017 at 2:12 AM, Daniel Jeliński wrote: > Hello, > I'm running an application doing a lot of Puts (size anywhere between 0 and > 10MB, one cell at a time); occasionally I'm getting an error like

Re: OutOfMemoryError: Direct buffer memory on PUT

Have you try to increase direct memory size for server process? -XXMaxDirectMemorySize=? On Mon, Oct 9, 2017 at 2:12 AM, Daniel Jeliński wrote: > Hello, > I'm running an application doing a lot of Puts (size anywhere between 0 and > 10MB, one cell at a time); occasionally

OutOfMemoryError: Direct buffer memory on PUT

Hello, I'm running an application doing a lot of Puts (size anywhere between 0 and 10MB, one cell at a time); occasionally I'm getting an error like the below: 2017-10-09 04:29:29,811 WARN [AsyncProcess] - #13368, table=researchplatform:repo_stripe, attempt=1/1 failed=1ops, last exception: