Re: TM OOMKilled

2022-02-15 Thread Xintong Song
> *From:* Xintong Song > *Sent:* Monday, February 14, 2022 10:06 PM > *To:* Alexey Trenikhun > *Cc:* Flink User Mail List > *Subject:* Re: TM OOMKilled > > Hi Alexey, > > You may want to double check if `state.backend.rocksdb.memory.managed` is > configure

Re: TM OOMKilled

2022-02-15 Thread Alexey Trenikhun
onday, February 14, 2022 10:06 PM To: Alexey Trenikhun Cc: Flink User Mail List Subject: Re: TM OOMKilled Hi Alexey, You may want to double check if `state.backend.rocksdb.memory.managed` is configured to `true`. (This should be `true` by default.) Another question that may or may not be relat

Re: TM OOMKilled

2022-02-14 Thread Xintong Song
fMemoryError, but because total container memory exceeds 10gb > > Thanks, > Alexey > -- > *From:* Caizhi Weng > *Sent:* Monday, February 14, 2022 6:42:05 PM > *To:* Alexey Trenikhun > *Cc:* Flink User Mail List > *Subject:* Re: TM OOMKille

Re: TM OOMKilled

2022-02-14 Thread Alexey Trenikhun
Sent: Monday, February 14, 2022 6:42:05 PM To: Alexey Trenikhun Cc: Flink User Mail List Subject: Re: TM OOMKilled Hi! Heap memory usage depends heavily on your job and your state backend. Which state backend are you using and if possible could you share your user code or explain what operations

Re: TM OOMKilled

2022-02-14 Thread Caizhi Weng
Hi! Heap memory usage depends heavily on your job and your state backend. Which state backend are you using and if possible could you share your user code or explain what operations your job is doing? Alexey Trenikhun 于2022年2月15日周二 05:17写道: > Hello, > We run Flink 1.13.5 job in app mode in Kube

TM OOMKilled

2022-02-14 Thread Alexey Trenikhun
Hello, We run Flink 1.13.5 job in app mode in Kubernetes, 1 JM and 1 TM, we also have Kubernetes cron job which takes savepoint every 2 hour (14 */2 * * *), once in while (~1 per 2 days) TM is OOMKilled, suspiciously it happens on even hours ~4 minutes after savepoint start (e.g. 12:18, 4:18) bu