Hi! Which API are you using? The datastream API or the Table / SQL API? If it is the Table / SQL API then some Java classes for some operators (for example aggregations, projection, filter, etc.) will be generated when compiling user code to executable Java code. These Java classes are new to the JVM. So if you're running too many jobs in the same Flink cluster a metaspace OOM might occur. There is already a JIRA ticket for this [1].
I don't know much about the behavior of class loaders, so I'll wait for others to apply in this aspect. [1] https://issues.apache.org/jira/browse/FLINK-15024 Puneet Duggal <puneetduggal1...@gmail.com> 于2021年9月13日周一 下午7:49写道： > Hi, > > So on going through multiple resources, got basic idea that JVM Metaspace > is used by flink class loader to load class metadata which is used to > create objects in heap. Also this is a one time activity since all the > objects of single class require single class metadata object in JVM > Metaspace. > > But while deploying multiple jobs on task manager, i saw almost linear > increase in consumption of metaspace (irrespective of parallelism). Even if > those multiple jobs have exactly same implementation. So wanted to confirm > if each job in flink has its own class loader which loads required classes > in Task Manager JVM Metaspace. > > PS: Any documentation for this will be of great help. > > Thanks, > Puneet
