[
https://issues.apache.org/jira/browse/HUDI-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17484836#comment-17484836
]
sivabalan narayanan commented on HUDI-2960:
-------------------------------------------
CC [[email protected]] [~rxu]
> create hudi table may cause memory leak in spark thrift server
> --------------------------------------------------------------
>
> Key: HUDI-2960
> URL: https://issues.apache.org/jira/browse/HUDI-2960
> Project: Apache Hudi
> Issue Type: Bug
> Components: spark
> Affects Versions: 0.10.0
> Reporter: suheng.cloud
> Priority: Major
>
> Hi, community
> I currently try to use spark-hudi integration in spark-thrift-server, and
> after test create hudi table for a while, I found it would finally result in
> META-SPACE OOM(in my case, jvm option -XX:MaxMetaspaceSize=256m assigned).
> After track the source, I found that every time a CreateHoodieTableCommand
> performed, `HiveClientUtils.newClientForMetadata` will be invoked, thus a
> IsolatedClientLoader will be created, in my scene, the OOM will occured after
> about 10 create statement executed.
> Why not use
> `sessionState.catalog.externalCatalog.asInstanceOf[ExternalCatalogWithListener].unwrapped.asInstanceOf[HiveExternalCatalog].client
> ` instead ? Does it has anything side effect?
> env: hudi master/spark-3.1.2/hive-2.3.6
> Thanks.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
