suheng.cloud created HUDI-2960:
----------------------------------
Summary: create hudi table may cause memory leak in spark thrift
server
Key: HUDI-2960
URL: https://issues.apache.org/jira/browse/HUDI-2960
Project: Apache Hudi
Issue Type: Bug
Components: Spark Integration
Affects Versions: 0.10.0
Reporter: suheng.cloud
Hi, community
I currently try to use spark-hudi integration in spark-thrift-server, and after
test create hudi table for a while, I found it would finally result in
META-SPACE OOM(in my case, jvm option -XX:MaxMetaspaceSize=256m assigned).
After track the source, I found that every time a CreateHoodieTableCommand
performed, `HiveClientUtils.newClientForMetadata` will be invoked, thus a
IsolatedClientLoader will be created, in my scene, the OOM will occured after
about 10 create statement executed.
Why not use
`sessionState.catalog.externalCatalog.asInstanceOf[ExternalCatalogWithListener].unwrapped.asInstanceOf[HiveExternalCatalog].client
` instead ? Does it has anything side effect?
env: hudi master/spark-3.1.2/hive-2.3.6
Thanks.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
