jerryshao commented on PR #8252: URL: https://github.com/apache/gravitino/pull/8252#issuecomment-3227409000
> My biggest concern is the maintainability of the clear resource logic. It seems that we are doing some complex work that should belong to the JVM level's GC, and this workload will get out of control as the project develops. > > I don't know if we have other alternative solutions to solve the OOM problem, but the current solution seems not to be the optimal one in the long run. WDYT? @jerryshao @FANNG1 I agree. My thought is that though we may have lots of catalogs, the number of providers is limited, which means the number of classloaders are limited, we don't have create a classloader for each catalog, we can reuse the classloader for the same type of catalogs, with this we can reduce the class-related OOM issues. Also, many engines like Spark/Trino that support multiple connectors will face the same problem, especially for Trino, they can also support dynamic connector register/unregister, how do they handle this problem? We'd better have a long term solution. For now, I think we can have a short solution, but we should clearly separate this from the main logic, so we can remove it easily. Also, we should add the comments as possible as we can. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
