Hi, what means „our state“ in this context? The total state in the backend or the state under one key? If you use, e.g. list state, I could see that the state for one key can grow above 2GB, but once we retrieve the state back from RocksDB as Java arrays (in your stacktrace, when making a checkpoint), which are bounded in size to a maximum of 2GB (Integer.MAX_VALUE) and maybe that is what happens in JNI if you try to go beyond that limit. Could that be a reason for your problem?
> Am 26.05.2017 um 15:50 schrieb Robert Metzger <[email protected]>: > > Hi Jason, > > This error is unexpected. I don't think its caused by insufficient memory. > I'm including Stefan into the conversation, he's the RocksDB expert :) > > On Thu, May 25, 2017 at 4:15 PM, Jason Brelloch <[email protected] > <mailto:[email protected]>> wrote: > Hey guys, > > We are running into a JVM crash on checkpointing when our rocksDB state > reaches a certain size on a taskmanager (about 2GB). The issue happens with > both a hadoop backend and just writing to a local file. > > We are running on Flink 1.2.1. > > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x00007febf4261b42, pid=1, tid=0x00007fead135f700 > # > # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build > 1.8.0_131-b11) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode > linux-amd64 compressed oops) > # Problematic frame: > # V [libjvm.so+0x6d1b42] jni_SetByteArrayRegion+0xc2 > # > # Core dump written. Default location: //core or core.1 > # > # An error report file with more information is saved as: > # /tmp/hs_err_pid1.log > # > # If you would like to submit a bug report, please visit: > # http://bugreport.java.com/bugreport/crash.jsp > <http://bugreport.java.com/bugreport/crash.jsp> > # > > Is this an issue with not enough memory? Or maybe not enough allocated to > rocksDB? > > I have attached the taskmanager logs, and the core dump. The jobmanager logs > just say taskmanger lost/killed. > > Thanks! > > -- > Jason Brelloch | Product Developer > 3405 Piedmont Rd. NE, Suite 325, Atlanta, GA 30305 > <http://www.bettercloud.com/> > Subscribe to the BetterCloud Monitor > <https://www.bettercloud.com/monitor?utm_source=bettercloud_email&utm_medium=email_signature&utm_campaign=monitor_launch> > - Get IT delivered to your inbox >
