Re: Flink 1.10 Out of memory

Okay, I think we need to close this thread. @Lasse Nedergaard - I hope your original question was answered by increase in JVM overhead. If not, please post again (cc me) and we will re-boot the discussion. @Som Lima - It is not nice to say bad things about other people here on the mailing list,

Re: Flink 1.10 Out of memory

@Zahir what the Apache means is don't be like Jesse Anderson who recommended Flink on the basis Apache only uses maps as seen in video. While Flink uses ValueState and State in Streaming API. Although it appears Jesse Anderson only looked as deep as the data stream helloworld. You are

Re: Flink 1.10 Out of memory

> " a simple Stack Overflow search yields. " Actually it wasn't stack overflow but a video I saw presented by Robert Metzger. of Apache Flink org. Your mind must have been fogged up with another thought of another email not the contents of my email clearly. He explained the very many

Re: Flink 1.10 Out of memory

@zahid I would kindly ask you to rethink you approach to posting here. Wanting to help answering questions is appreciated, but what you post is always completely disconnected from the actual issue. The questions here usually go beyond the obvious and beyond what a simple Stack Overflow search

Re: Flink 1.10 Out of memory

https://youtu.be/UEkjRN8jRx4 22:10 - one option is to reduce flink managed memory from default 70% to may be 50%. - This error could be caused also due to missing memory ; - maintaining a local list by programmer so over using user allocated memory caused by heavy processing ; - or

Re: Flink 1.10 Out of memory

True. Thanks for the clarification. Thank you~ Xintong Song On Fri, Apr 24, 2020 at 5:21 PM Stephan Ewen wrote: > I think native methods are not in a forked process. It is just a malloc() > call that failed, probably an I/O buffer or so. > This might mean that there really is no native

Re: Flink 1.10 Out of memory

I think native methods are not in a forked process. It is just a malloc() call that failed, probably an I/O buffer or so. This might mean that there really is no native memory available any more, meaning the process has hit its limit. In any case, a bit more JVM overhead should solve this. On

Re: Flink 1.10 Out of memory

I might be wrong about how JNI works. Isn't a native method always executed in another process? I was searching for the java error message "Cannot allocate memory", and it seems this happens when JVM cannot allocate memory from the OS. Given the exception is thrown from calling a native method, I

Re: Flink 1.10 Out of memory

@Xintong - out of curiosity, where do you see that this tries to fork a process? I must be overlooking something, I could only see the native method call. On Fri, Apr 24, 2020 at 4:53 AM Xintong Song wrote: > @Stephan, > I don't think so. If JVM hits the direct memory limit, you should see the

Re: Flink 1.10 Out of memory

@Stephan, I don't think so. If JVM hits the direct memory limit, you should see the error message "OutOfMemoryError: Direct buffer memory". Thank you~ Xintong Song On Thu, Apr 23, 2020 at 6:11 PM Stephan Ewen wrote: > @Xintong and @Lasse could it be that the JVM hits the "Direct Memory" >

Re: Flink 1.10 Out of memory

@Xintong and @Lasse could it be that the JVM hits the "Direct Memory" limit here? Would increasing the "taskmanager.memory.framework.off-heap.size" help? On Mon, Apr 20, 2020 at 11:02 AM Zahid Rahman wrote: > As you can see from the task manager tab of flink web dashboard > > Physical

Re: Flink 1.10 Out of memory

As you can see from the task manager tab of flink web dashboard Physical Memory:3.80 GB JVM Heap Size:1.78 GB Flink Managed Memory:128 MB *Flink is only using 128M MB which can easily cause OOM* *error.* *These are DEFAULT settings.* *I dusted off an old laptop so it only 3.8 GB RAM.* What

Re: Flink 1.10 Out of memory

Hi Thnaks for the reply. We Will try it out and let everybody know Med venlig hilsen / Best regards Lasse Nedergaard > Den 20. apr. 2020 kl. 08.26 skrev Xintong Song : > >  > Hi Lasse, > > From what I understand, your problem is that JVM tries to fork some native > process (if you look at

Re: Flink 1.10 Out of memory

https://betsol.com/java-memory-management-for-java-virtual-machine-jvm/ Backbutton.co.uk ¯\_(ツ)_/¯ ♡۶Java♡۶RMI ♡۶ Make Use Method {MUM} makeuse.org On Fri, 17 Apr 2020 at 14:07, Lasse Nedergaard < lassenedergaardfl...@gmail.com> wrote: > Hi. > > We have migrated

Flink 1.10 Out of memory

Hi. We have migrated to Flink 1.10 and face out of memory exception and hopeful can someone point us in the right direction. We have a job that use broadcast state, and we sometimes get out memory when it creates a savepoint. See stacktrack below. We have assigned 2.2 GB/task manager and