Hi Yuxia, I did increase the parallelism to 16 but that is causing memory overflowing issues. Task manager heap memory collapses after a certain point when the job has run. I'm attaching the metrics, the flatmap converts jsons and parses them to comma separated strings. Could you suggest how to optimize it?
[image: image.png] On Fri, May 20, 2022 at 2:39 PM yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > HI, I think you can increase the parallelism of the flat map operator. For > SQL job, you can refer the doc[1] to set parallelism. For datastream job, > you can set the parallelism in your code. > > > Also, if possible, you can try optimize your code in the flatmap node . > > [1]: > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-resource-default-parallelism > > Best regards, > Yuxia > > ------------------------------ > *发件人: *"Zain Haider Nemati" <zain.hai...@retailo.co> > *收件人: *"User" <u...@flink.apache.org>, "dev" <dev@flink.apache.org> > *发送时间: *星期五, 2022年 5 月 20日 下午 4:51:14 > *主题: *Flatmap node at 100% > > Hi, > Im seeing this behaviour in my flink job, what can I do to remove this > bottleneck > > [image: image.png] > >
