Re: Correlation between number of operators and Job manager memory requirements

Hi Till, Thanks for your reply. >> My suggestion would be to split the different patterns up and run them with in different jobs. I'm not able to understand how splitting up the jobs based on patterns would be more efficient than based on the key. The total number of operators would still be

Re: Correlation between number of operators and Job manager memory requirements

Hi Shailesh, I fear that given your job topology, it is not that surprising that things break. The problem is that you might have M x N CEP operators concurrently active. This means that they have to keep their state in memory. Given 3.5 GB isn't that much if you have more than 300 CEP NFAs

Re: Correlation between number of operators and Job manager memory requirements

Hi Till, When I'm submitting one big job, both JM and TM (sometimes just JM) are crashing at the time of initialization itself (i.e. not all operators switch to RUNNING) with OOM. The number of threads on TM go to almost 1000. But when I'm submitting multiple jobs, job submission is completed.

Re: Correlation between number of operators and Job manager memory requirements

Hi Shailesh, my question would be where do you see the OOM happening? Does it happen on the JM or the TM. The memory requirements for each operator strongly depend on the operator and it is hard to give a general formula for that. It mostly depends on the user function. Flink itself should not

Re: Correlation between number of operators and Job manager memory requirements

Actually, there are too many hyperparameters to experiment with, that is why I'm trying to understand if there is any particular way in which a cluster could be benchmarked. Another strange behaviour I am observing is: Delaying the operator creation (by distributing the operators across jobs, and

Re: Correlation between number of operators and Job manager memory requirements

Hi, You could definitely try to find formula for heap size, but isnt's it easier just to try out different memory settings and see which works best for you? Thanks, Pawel 17 lut 2018 12:26 "Shailesh Jain" napisał(a): Oops, hit send by mistake. In the

Re: Correlation between number of operators and Job manager memory requirements

Oops, hit send by mistake. In the configuration section, it is mentioned that for "many operators" heap size should be increased. "JVM heap size (in megabytes) for the JobManager. You may have to increase the heap size for the JobManager if you are running very large applications (with many

Correlation between number of operators and Job manager memory requirements

Hi, I have flink job with almost 300 operators, and every time I'm trying to submit the job, the cluster crashes with OutOfMemory exception. I have 1 job manager and 1 task manager with 2 GB heap space allocated to both. In the configuration section of the documentation