In Flink, you can only limit memory usage, e.g. via taskmanager.memory.process.size [1] (throttling could be implemented using the DataStream API, but you mentioned you are using SQL). Quotas on other resources can be set in the underlying resource manager.
But I'd suggest investigating the failure and understand what's causing it. Probably, high resource usage is not the root cause. [1] https://ci.apache.org/projects/flink/flink-docs-stable/deployment/config.html#memory-configuration Regards, Roman On Mon, Apr 12, 2021 at 10:17 AM 张颖 <queyue...@163.com> wrote: > > Hi, > This is not my intention. > I was meaning that I run stream jobs and batch jobs in the same cluster, but > the batch job almost preemption all the resource in the cluster(maybe lead to > the machine loadaveage to 150 or cpu to 100% or disk io to 100%), which lead > my steam job to a series of problems (such as tm lost and connection time > out). So I want wo limit the speed of processing data on batch job. > > > > > > > > At 2021-04-12 15:49:31, "Roman Khachatryan" <ro...@apache.org> wrote: > >Hi, > > > >I'm not sure that I fully understand your question. > >Is the intention to prioritize some jobs over the others in the same > >Flink cluster? Currently, it is not possible (FLIP-156 and further > >work aim to address this [1]). At the moment, you can either > >- deploy the jobs in separate clusters (per-job mode [2]) and rely on > >the underlying resource manager for resource isolation > >- or allocate less task slots to a lower priority job by configuring: > >parallelism, operator chaining and slot sharing groups > > > >[1] > >https://cwiki.apache.org/confluence/display/FLINK/FLIP-156%3A+Runtime+Interfaces+for+Fine-Grained+Resource+Requirements > >[2] > >https://ci.apache.org/projects/flink/flink-docs-stable/deployment/#per-job-mode > > > >Regards, > >Roman > > > > > > > >On Mon, Apr 12, 2021 at 9:07 AM 张颖 <queyue...@163.com> wrote: > >> > >> When I run a sql job with blink planner in my cluster,the task is almost > >> preemption the whole resources in the cluster, and this is a bad effect > >> to the stream task.As it is not necessary on speed,so is there any way to > >> control the rate in my batch task? > >> > >> > >> > >> this is the machine performance in running some operator: > >> https://issues.apache.org/jira/browse/FLINK-22204 > >> > >> > >> > >> > > > >
